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Diener,  David  Alan.  Ph.D.,  Purdue  University,  December  1989. 
Forecasting  Air  Base  Operability  in  a  Hostile  Environment: 
Estimating  Metamodels  from  Large-Scale  Simulations. 

Co-chairs:  Robert  D.  Plante  and  James  R.  Wilson. 

An  on-going  Air  Force  logistics  concern  is  the  ability 
of  an(Air  Force/ unit  to  fly  aircraft  into  combat  particularly 
when  their  air  base  comes  under  attack.  Air  bases  are  no 
longer  sanctuaries;  Air  Force 'units  must  not  only  survive 
attacks  but  continue  to  operate  afterwards  as  well.  Limited 
budgets  and  long  procurement  and  training  pipelines  magnify 
the  problem,  making  it  imperative  to ^specif ically^ identify 
and  resolve  support  system  deficiencies.  A  systems  view  of 
tne  support  structure  rather  than  narrow  functional  views  is 
essential.  "'We  propose  a  simulation  approach  to  the  problem 
which  attempts  to  capture  the  logistics  infrastructure  for  a 
single  air  base.  Multiple  simulation  runs  are  used  to  derive 
a  simpler  metamodel  useful  for  forecasting  future  performance 
or  for  evaluating  policy  alternatives.  This  metamodel  can 
then  be  used- in  lieu  of  complex  and  costly  simulation  models 
to  explore  "wr.at  if"  analyses.  'Ma  j  o r-  research  issues  include 
eSv.imai.ion  or  metamodels  from  larce  — s^ale  simulation 
i..odeis  with  nigmy  correlated  responses, ,  experimental  design 


'with  simular  responses  and  application  of  variance  reduction 
techniques  to  large-scale  simulation  problems.  Good  variance 
reduction  results  are  obtained  using  a  classical  two-level 
experimental  design  with  blocking  within  the  fraction  based 
on  common  random  numbers.  Two  cases,  with  and  without 
attacks  on  the  air  base,  are  modeled  as  sub-experiments. 
Results  indicate  homogeneity  of  variance  within  each  case, 
but  heterogeneity  between  the  two  cases.  A  significant 
difference  in  the  number  of  sorties  flown  when  the  air  base 
is  attacked  is  evident,  but  this  difference  dissipates  by  Day 
30.  The  estimated  metamodels  indicate  that  two-way 
interactions  are  extremely  important  and  should  not  be 
ignored.  Also  the  daily  metamodels  differ  not  only  in  the 
slopes  of  the  factors,  but  also  in  the  very  factors  found  to 
be  significant.  Many  interesting  insights  into  the 
complexities  and  interdependencies  of  the  factors  defining 
the  logistics  infrastructure  are  highlighted  in  this 
research . 
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To  the  men  and  women  who  daily  prepare  and  study  for 
war,  not  for  its  glorification  or  continuation,  but  for  its 
prevention  until  the  day 

\  They  will  beat  their  swords  into 

plowshares  and  their  spears  into  pruning 
hooks.  Nation  will  not  take  up  sword 
against  nation,  nor  will  they  train  for 
war  anymore.  Every  man  will  sit  under 
his  own  vine  and  under  his  own  fig  tree, 
and  no  one  will  make  them  afraid. 

(Micah  4:3-4) 


) 


Ill 


ACKNOWLEDGMENTS 


Many  people  have  contributed  to  the  success  of  this 
dissertation.  Members  of  my  committee  have  contributed  in 
different  ways.  Dr.  Timothy  J.  Lowe  always  encouraged  me  and 
believed  in  me  from  the  very  start  of  my  doctoral  program  and 
I  thank  him  for  his  support  and  example.  The  enthusiasm  of 
Dr.  James  R.  Wilson  for  simulation  and  academic  excellence 
will  always  be  remembered.  His  interest  and  guidance  in 
researching  this  particular  problem  were  invaluable.  I  thank 
him  for  his  expert  guidance  with  the  analysis  and  for  his 
many  hours  of  wrestling  with  the  TSAR  computer  code  so  we 
could  run  the  model  at  Purdue.  I  thank  Dr.  Robert  D.  Plante 
for  introducing  me  to  many  new  ideas  about  experimental 
design  and  for  whetting  my  appetite  for  research  in  that 
area.  I  also  extend  my  thanks  to  the  RAND  Corporation  for 
providing  the  TSAR/TSARINA  models  and  databases.  Mr.  Don  E. 
Emerson  and  Dr.  John  R.  Folkeson  of  RAND  willing  contributed 
many  hours  of  their  personal  time  to  help  get  the  models 
running  at  Purdue  University  so  that  I  could  complete  my 
research . 

I  also  want  to  especially  recognize  Dr.  Folkeson  for  his 
role  in  my  education  over  the  years.  He  was  the  first  to 


IV 


encourage  me  to  pursue  a  doctorate;  this  research  stems  from 
some  ideas  and  work  we  brainstormed  five  years  ago  in  a 
remote  corner  of  the  Pentagon.  I  highly  regard  his  example 
as  a  scholar  and  gentleman. 

My  parents  have  also  contributed  tremendously  to  my 
being  able  to  complete  this  doctorate.  My  father,  Lt .  Col. 
William  E.  Diener,  instilled  in  me  from  ’’Day  1"  to  always  do 
my  best  in  all  that  I  attempt .  From  him  I  have  learned  that 
sincere  effort,  dedication,  and  perseverance  can  many  times 
make  up  for  what  I  lack  in  talent.  My  mother,  Jean  L. 

Diener,  taught  me  to  be  an  individual,  one  with  high  personal 
standards  and  goals,  not  content  with  being  mediocre.  My 
sisters,  Dianne,  Denise,  and  Dorinda  are  also  thanked  for 
their  many  words  of  encouragement. 

My  wife  and  children  have  continuously  supported  me  in 
accomplishing  this  task.  Ann  is  the  delight  of  my  every  day 
who  never  doubted  that  I  would  finish  this  and  made  many 
sacrifices  so  that  I  could.  Derek  and  Karin  often  had  to  be 
without  their  Dad  so  he  could  "finish  writing  his  book."  I 
thank  them  for  being  quiet  over  the  years  so  I  could  work. 

Finally,  I  thank  my  Lord  Jesus  Christ  for  the  blessings 
and  talents  He  has  provided  that  made  this  work  possible. 

This  is  intended  for  His  glory,  not  mine.  To  Him  be  the 
glory! 


V 


TABLE  OF  CONTENTS 


Page 

LIST  OF  TABLES .  ix 

LIST  OF  FIGURES .  xi 

ABSTRACT .  xvi 

CHAPTER  I  -  THE  AIR  BASE  OPERABILITY  PROBLEM  .  1 

Introduction  .  1 

Research  Objectives  .  4 

Organization  of  The  Thesis  .  9 

CHAPTER  II  -  BACKGROUND  AND  LITERATURE  REVIEW  .  11 

The  Sortie  Generation  Process  .  H 

Simulation  and  Analytic  Tools  for  Modeling 

Air  Bases  .  15 

Simulation  Models  .  16 

An  Analytic  Model  .  18 

Current  ABO  Research .  18 

A  Previous  Study  of  an  Air  Base  Logistics 

Infrastructure  .  18 

The  Logistics  Variables  .  20 

The  Experimental  Design  and  Model  .  20 

Findings  and  Conclusions  .  21 

Problems  With  the  Design  .  21 

Statistically-Controlled  Experimental  Design  .  22 

Variance  Reduction  Techniques  In  Simulation  ...  23 

Random  Number  Assignment  in 

Monte  Carlo  Simulation  .  24 

CHAPTER  III  -  METHODOLOGY .  27 

The  Resource  Factors  .  27 

Factor  B:  Filler  or  Replacement  Aircraft  ......  29 

Factor  C:  Aircraft  Battle  Damage  Repair 

( ABDR)  Capability  .  32 

Factor  D:  Recovery  From  Air  Base  Attack .  33 

Factor  E:  Maintenance  Personnel  .  34 


VI 


Page 

Factor  F:  Avionics  Intermediate  Test 

Station  (AIS)  .  35 

Factor  G:  Support  Equipment  .  35 

Factor  H:  Spare  Parts  .  36 

Factor  J:  Missiles  .  37 

Factor  K:  Fuel  .  37 

The  Environmental  Factors  .  38 

Air  Base  Attacks  .  39 

Attrition  of  Our  Aircraft  .  39 

Air  Tasking  Orders  (ATO)  .  41 

The  Metamodels  to  be  Estimated  .  42 

The  Metamodel  for  .the  Case  of  No  Attacks  .  42 

The  Attack  Case  Metamodel  .  43 

The  Experimental  Design  .  44 

The  Revised  Metamodels  (With  Blocking)  .  49 

The  Metamodel  for  the  No  Attack  Case  .  4  9 

The  Attack  Case  Metamodel  .  31 

Assessing  the  Research  Objectives  .  52 

The  Effectiveness  of  the  Experimental  Design  . .  52 

Estimating  the  Daily  Metamodels  .  53 

Evaluating  the  Impact  of  Attacks  .  59 

Identifying  Key  Resources  Over  Time  .  65 

CHAPTER  IV  -  OVERALL  EVALUATION  OF  THE 

EXPERIMENTAL  DESIGN  .  67 

The  Effectiveness  of  the  Experimental  Design  .  67 

Variance  Reduction  and  Design  Efficiency  .  68 

The  Daily  Metamodels  .  74 

Coding  of  the  Design  Matrix  .  74 

Test  for  Normality  .  7  6 

Statistics  For  Design  Point  k  .  81 

Metamodels  for  the  No-Attack  Case  .  87 

Metamodels  for  the  Attack  Case  .  95 

The  Impact  of  Attacks  .  104 

CHAPTER  V  -  INTERPRETATION  OF  THE  METAMODELS .  110 

Overall  Comparison  of  the  No-Attack 

and  Attack  Cases  .  110 

Attack  Versus  No-Attack  .  Ill 

All  Factors  High  Versus  All  Factors  Low  .  116 

Summary:  All  Factors  High  Versus  All  Factors 

Low  and  Attack  Versus  No-Attack  .  119 

No-Attack  Case  Factor  Results  .  119 

Main  Effect:  Filler  or  Replacement  Aircraft  ...  123 

Main  Effect:  Personnel  .  131 

Main  Effect:  Spare  Parts  .  135 

Main  Effect:  Missiles  .  138 

Main  Effect:  Fuel  .  143 


Vll 


Page 


Main  Effect:  ABDR .  148 

Other  Main  Effects:  AIS,  Attrition, 

Support  Equipment,  and  Recovery  .  14  8 

Attack  Case  Factor  Results  .  156 

Main  Effect:  Recovery  Resources  .  159 

Main  Effect:  Personnel  .  161 

Main  Effect:  Fuel  .  163 

Main  Effect:  ABDR .  167 

Main  Effect:  Attrition  .  172 

Main  Effect:  Filler  or  Replacement  Aircraft  ...  175 

Main  Effect:  Missiles  .  178 

Main  Effect:  Support  Equipment  (SE)  .  181 

Other  Main  Effects:  Spares  and  AIS  .  184 

Key  Resources  Over  Time  .  187 

Factor  A:  Attrition  .  191 

Factor  B:  Filler  Aircraft  .  193 

Factor  C:  ABDR . • .  193 

Factor  D :  Recov  ^ry  Resources  . .  196 

Factor  E:  Personnel  .  196 

Factor  F:  AIS  Test  Sets  .  199 

Factor  G:  Support  Equipment  .  201 

Factor  H:  Spare  Parts  .  203 

Factor  J:  Missiles  .  203 

Factor  K:  Fuel  .  206 

Overall  Importance  of  Factors  .  208 

Attack  Case  .  208 

No-Attack  Case  .  212 

CHAPTER  VI  -  CONCLUSIONS  AND  FUTURE  RESEARCH  .  217 

Scope  .  217 

Conclusions  .  218 

Research  Objective  1:  Experimental  Design  .  218 

Research  Objective  2:  Estimating  Metamodels  ...  220 

Research  Objective  3:  Impact  of  Attacks  .  221 

Research  Objective  4:  Key  Resources  .  224 

Limitations  .  226 

Future  Research  .  227 

Variance  Reduction  .  228 

Widening  the  Inference  Space  .  231 

Performance  Measures  .  ... .  233 

LIST  OF  REFERENCES .  240 

APPENDICES 

Appendix  A:  Equivalency  of  Model  Forms  .  244 

Appendix  B:  Regression  Results  for  No-Attack  Case  ..  247 

Appendix  C:  Residual  Results  for  No-Attack  Case  ....  277 

Appendix  D:  Regression  Results  for  Attack  Case  .  307 


Vlll 


Page 

Appendix  E:  Residual  Results  for  Attack  Case  .  337 

VITA .  367 


') 


ix 


LIST  OF  TABLES 

Table  Page 

3.1  High  Levels  for  the  Resource  Factors  .  3Q 

3.2  Low  Levels  for  the  Resource  Factors  .  31 

3.3  Summary  of  the  Air  Base  Attacks  . 

4.1  No-Attack  Case  —  Variance  Reduction  and 

Relative  Efficiency  Due  to  Blocking  .  69 

4.2  Attack  Case  —  Variance  Reduction  and  Relative 

Efficiency  Due  to  Blocking  .  70 

4.3  No-Attack  Case  —  Computer  Run  Times 

(Seconds)  .  71 

4.4  Attack  Case  -  Computer  Run  Times  (Seconds)  .  73 

4.5  Sample  Statistics  and  Shapiro-Wilk 

Test  Results  .  77 

4.6  Confidence  Intervals  for  |lk(i)  in  the 

No-Attack  Case  .  82 

•k 

4.7  Confidence  Intervals  for  Jl^d)  in  the 

Attack  Case  .  83 

4.8  Results  for  Homogeneity  of  Variance  Tests  .  86 

4 . 9  No-Attack  Case  --  Daily  Metamodels  .  88 

4.10  No-Attack  Case  —  Regression  Statistics  .  94 

4.11  Attack  Case  --  Daily  Metamodels  .  96 

4.12  Attack  Case  --  Regression  Statistics  .  103 

4.13  Tests  for  Significant  Impact  of  Attacks 


106 


X 


Table  Page 

4.14  Confidence  Intervals  for  Differences 

Attributed  to  Attacks  .  108 

5.1  Net  Sorties  Over  30-Days  for  Factor  at  High  Level 

Versus  Low  Level  .  190 


xi 


LIST  OF  FIGURES 


Figure  Page 

1.1  TSAR/TSARINA  for  Analyzing  Sortie 

Generation  Capability  .  5 

1.2  Single  Response  Simulation  Analysis  .  7 

1.3  Air  Base  Operability  Framework  .  8 

2.1  Sortie  Generation  Process  .  12 

2.2  Sortie  Generation  Dynamics  .  13 

3.1  TSAR/TSARINA  Database  Factors  .  28 

3.2  The  Experimental  Design  .  45 

3.3  The  Blocking  Scheme  .  50 

4.1  Normal  Quantile  Plot  —  All  Factors  High, 

Attack  Case,  Day  5  .  78 

4.2  Normal  Quantile  Plot  —  All  Factors  High, 

No-Attack  Case,  Day  5  .  79 

4.3  Normal  Quantile  Plot  —  All  Factors  Low, 

Attack  Case,  Day  1  .  SO 

4.4  Confidence  Intervals  for  Mean  Responses  .  85 

4.5  Confidence  Intervals  for  Differences 

Attributed  to  Attacks  .  109 

5.1  Sorties  Per  Day  with  All  Factors  Low  .  112 

5.2  Difference  Between  Attack  and  No-Attack 

Cases  with  All  Factors  Low  .  113 

5.3  Sorties  Per  Day  with  All  Factors  High  .  114 


xii 

Figure  Page 

5.4  Difference  Between  Attack  and  No-Attack  Cases 

with  All  Factors  High  .  H4 

5.5  No-Attack  Case  —  All  Factors  High  Versus 

All  Factors  Low  . H7 

5.6  No-Attack  Case  —  Difference  Between  All 

Factors  High  and  All  Factors  Low  .  118 

5.7  Attack  Case  —  All  Factors  High  Versus 

All  Factors  Low  .  120 

5.8  Attack  Case  —  Difference  Between  All  Factors 

High  and  All  Factors  Low  .  121 

5.9  Comparison  of  Four  Resource-Attack 

Combinations  .  122 

5.10  No-Attack  Case  --  Significant  Main  Factors  (1)  ..  124 

5.11  No-Attack  Case  —  Significant  Main  Factors  (2)  ..  125 

5.12  No-Attack  Case  —  Filler  Aircraft 

Interactions  (1)  .  127 

5.13  No-Attack  Case  —  Filler  Aircraft 

Interactions  (2)  129 

5.14  No-Attack  Case  —  Overall  Contribution  of 

Filler  Aircraft  .  130 

5.15  No-Attack  Case  —  Personnel  Interactions  .  133 

5.16  No-Attack  Case  —  Overall  Contribution 

of  Personnel  .  134 

5.17  No-Attack  Case  --  Spare  Part  Interactions  .  136 

5.18  No-Attack  Case  —  Overall  Contribution  of 

Spare  Parts  .  139 

5.19  No-Attack  Case  —  Missile  Interactions  (1)  .  140 

5.20  No-Attack  Case  —  Missile  Interactions  (2)  .  142 

5.21  No-Attack  Case  —  Overall  Contribution 

of  Missiles  .  144 

5.22  No-Attack  Case  —  Fuel  Interactions  .  145 


) 


Xlll 


Page 


Figure 

5.23 

5.24 

5.25 

5.26 

5.27 

5.28 

5.29 

5.30 

5.31 

5.32 

5.33 

5.34 

5.35 

5.36 

5.37 

5.38 

5.39 

5.40 

5.41 

5.42 

5.43 

5 .44 


No-Attack  Case  —  Overall  Contribution  of  Fuel .  .  147 

No-Attack  Case  --  ABDR  Interactions  .  14  9 

No-Attack  Case  —  Overall!  Contribution  of  ABDR..  150 

No-Attack  Case  —  Overall  Contribution  of  AIS  .  .  152 

No-Attack  Case  —  Overall  Contribution  of 

Attrition  .  153 

No-Attack  Case  —  Overall  Contribution  of 

Support  Equipment  .  154 

No-Attack  Case  --  Overall  Contribution  of 

Recovery  Resources  .  155 

Attack  Case  —  Significant  Main  Factors  (1)  ....  157 

Attack  Case  —  Significant  Main  Factors  (2)  ....  158 

Attack  Case  —  Overall  Contribution  of 

Recovery  Resources  .  160 

Attack  Case  —  Personnel  Interactions  .  162 

Attack  Case  —  Overall  Contribution  of 

Personnel  .  164 

Attack  Case  —  Fuel  Interactions  (1)  .  .  .  . .  165 

Attack  Case  —  Fuel  Interactions  (2)  166 

Attack  Case  —  Overall  Contribution  of  Fuel  ....  168 

Attack  Case  --  A3DR  Interactions  .  170 

Attack  Case  --  Overall  Contribution  of  ABDR  ....  171 

Attack  Case  --  Attrition  Interactions  .  173 

Attack  Case  —  Overall  Contribution  of 

Attrition  .  174 

Attack  Case  --  Filler  Aircraft  Interactions  ....  176 

Attack  Case  —  Overall  Contribution  of 

Filler  Aircraft  .  177 

Attack  Case  --  Missile  Interactions  .  179 


Page 


Figure 

5.45  Attack  Case  —  Overall  Contribution 

of  Missiles  .  180 

5.46  Attack  Case  —  Support  Equipment  Interactions  ..  182 

5.47  Attack  Case  —  Overall  Contribution  of 

Support  Equipment  .  183 

5.48  Attack  Case  —  Spare  Part  Interactions  .  185 

5.49  Attack  Case  —  Overall  Contribution  of 

Spare  Parts  .  186 

5.50  Attack  Case  AIS  Interactions  .  188 

5.51  Attack  Case  —  Overall  Contribution  of  AIS  .  189 

5.52  Overall  Contribution  of  Attrition  — 

Attack  Versus  No-Attack  .  182 

5.53  Overall  Contribution  of  Filler  Aircraft  — 

Attack  Versus  No-Attack  .  194 

5.54  Overall  Contribution  of  ABDR  — 

Attack  Versus  No-Attack  .  195 

5.55  Overall  Contribution  of  Recovery  Resources  — 

Attack  Versus  No-Attack  .  197 

5.56  Overall  Contribution  of  Personnel  — 

Attack  Versus  No-Attack  . . .  198 

5.57  Overall  Contribution  of  AIS  -- 

Attack  Versus  No-Attack  .  200 

5.58  Overall  Contribution  of  Support  Equipment  — 

Attack  Versus  No-Attack  .  202 

5.59  Overall  Contribution  of  Spare  Parts  — 

Attack  Versus  No-Attack  .  204 

5.60  Overall  Contribution  of  Missiles  — 

Attack  Versus  No-Attack  .  205 

5.61  Overall  Contribution  of  Fuel  — 

Attack  Versus  No-Attack  .  207 

5.62  Attack  Case  —  Overall  Contribution  of 

Five  Factors  (1)  .  209 


XV 


Figure  Page 

5.63  Attack  Case  —  Overall  Contribution  of 

Five  Factors  (2)  210 

5.64  No-Attack  Case  —  Overall  Contribution  of 

Five  Factors  (1)  213 

5.65  No-Attack  Case  —  Overall  Contribution  of 

Five  Factors  (2)  214 

5.66  No-Attack  Case  —  Overall  Contribution  of 

Five  Factors  (2)  —  Modified  .  215 

6.1  Comparison  of  Effects  of  Attacks  and 

Resource  Levels  .  223 

6.2  Multivariate  Stochastic  Model  .  237 

6.3  Response  to  an  Attack  as  an  Intervention  .  238 


ABSTRACT 


Diener,  David  Alan.  Ph.D.,  Purdue  University,  December  1989. 
Forecasting  Air  Base  Operability  in  a  Hostile  Environment: 
Estimating  Metamodels  from  Large-Scale  Simulations. 

Co-chairs:  Robert  D.  Plante  and  James  R.  Wilson. 

An  on-going  Air  Force  logistics  concern  is  the  ability 
of  an  Air  Force  unit  to  fly  aircraft  into  combat  particularly 
when  their  air  base  comes  under  attack.  Air  bases  are  no 
longer  sanctuaries;  Air  Force  units  must  not  only  survive 
attacks  but  continue  to  operate  afterwards  as  well.  Limited 
budgets  and  long  procurement  and  training  pipelines  magnify 
the  problem,  making  it  imperative  to  specifically  identify 
and  resolve  support  system  deficiencies.  A  systems  view  of 
the  support  structure  rather  than  narrow  functional  views  is 
essential.  We  propose  a  simulation  approach  to  the  problem 
which  attempts  to  capture  the  logistics  infrastructure  for  a 
single  air  base.  Multiple  simulation  runs  are  used  to  derive 
a  simpler  metamodel  useful  for  forecasting  future  performance 
or  for  evaluating  policy  alternatives.  This  metamodel  can 
then  be  used  in  lieu  of  complex  and  costly  simulation  models 
to  explore  "what  if"  analyses.  Major  research  issues  include 
the  estimation  of  metamodels  from  large-scale  simulation 
models  with  highly  correlated  responses,  experimental  design 


XVII 


with  simular  responses  and  application  of  variance  reduction 
techniques  to  large-scale  simulation  problems.  Good  variance 
reduction  results  are  obtained  using  a  classical  two-level 
experimental  design  with  blocking  within  the  fraction  based 
on  common  random  numbers.  Two  cases,  with  and  without 
attacks  on  the  air  base,  are  modeled  as  sub-experiments . 
Results  indicate  homogeneity  of  variance  within  each  case, 
but  heterogeneity  between  the  two  cases.  A  significant 
difference  in  the  number  of  sorties  flown  when  the  air  base 
is  attacked  is  evident,  but  this  difference  dissipates  by  Day 
30.  The  estimated  metamodels  indicate  that  two-way 
interactions  are  extremely  important  and  should  not  be 
ignored.  Also  the  daily  metamodels  differ  not  only  in  the 
slopes  of  the  factors,  but  also  in  the  very  factors  found  to 
be  significant.  Many  interesting  insights  into  the 
complexities  and  interdependencies  of  the  factors  defining 
the  logistics  infrastructure  are  highlighted  in  this 


research . 
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CHAPTER  I  -  THE  AIR  BASE  OPERABILITY  PROBLEM 

Introduction 

Air  "force,"  as  viewed  by  then  Assistant  Secretary  of 
the  Air  Force,  Tidal  McCoy,  consists  of  three  elements:  the 
air  base  from  which  aircraft  launch  and  recover,  the  aircraft 
themselves,  and  the  munitions  that  make  the  flying 
productive.  All  three  are  important  and  must  be  present  for 
air  "force"  to  exist.  In  recent  years,  renewed  emphasis  has 
centered  on  the  air  base  itself  and  air  base  operability 
(ABO) .  ABO  consists  of  interrelated  and  mutually  supporting 
tasks:  to  defend,  to  survive,  to  recover,  and  to  continue  to 

fly  aircraft  (McCoy,  1987) .  Here  we  will  focus  on  the  last 
two  tasks  -  to  recover  and  continue  to  fly. 

Air  bases  must  be  recognized  as  systems  vulnerable  to 
attacks  which  cause  substantial  resource  losses.  To  ensure 
support  for  combat  flying  operations,  managers  must  identify 
and  resolve  support  system  deficiencies.  Given  a  support 
system  view,  what  are  the  key  resources  and/or  capabilities? 
Which  are  susceptible  and  sensitive  to  attacks?  Questions 
such  as  these  must  be  answered  within  an  analysis  framework 
which  considers  the  interdependencies  of  the  resources  and 
functions  comprising  the  logistics  system  of  the  air  base. 
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Additionally,  we  must  have  a  method  to  measure  and  compare 
different  alternatives.  A  logical  choice  for  a  response 
variable  is  the  number  of  sorties  flown  per  day.  A  sortie  is 
the  flight  of  a  single  aircraft;  the  process  of  preparing  an 
aircraft  to  fly  is  called  sortie  generation.  The  number  of 
sorties  flown  per  day  (or  some  function  of  it)  is  a  measure 
of  how  effectively  the  flying  operation  is  proceeding  and 
thus  how  well  the  logistics  system  works  in  support  of  the 
flying  operations. 

Typically  the  air  base  resourcing  problem  has  been 
addressed  independently  by  diverse  functional  areas  while 
assuming  other  resources  remain  constant.  For  example,  past 
efforts  have  been  conducted  by  medical  and  personnel  planners 
to  predict  the  number  of  technicians  (by  specialty)  required 
to  fly  certain  sortie  rates.  Similarly  fuel  planners  have 
examined  the  fuel  requirements  (quantities,  storage  needs, 
deliveries,  etc.)  necessary  to  meet  wartime  sortie  rates. 
Similar  efforts  could  be  cited  for  munitions,  aircraft, 
support  equipment  and  so  on.  In  such  examinations  of 
individual  resource  areas,  no  interdependencies  are  usually 
assumed.  Approaches  dealing  with  specific  Air  Force  units 
have  focused  almost  exclusively  on  the  supply  or  spare  part 
resource  area  (see  Hillestad  1982,  Pyles  and  Tripp  1982) . 
While  these  include  several  of  the  major  components  of  the 
logistics  infrastructure,  they  still  have  a  rather  narrow 
focus  and  lack  a  total  logistics  system  perspective. 


3 


Rich  et  al.  (1987)  discuss  the  need  to  focus  on  sortie 
generation  in  rapidly  changing  war  environments  and  the 
importance  of  a  total  system  viewpoint.  The  problem  of 
projecting  force  capability  in  such  stressful  environments 
has  taken  two  separate  modeling  approaches  (simulation  and 
analytic) ,  each  with  its  own  inherent  shortcomings .  In 
brief,  the  limitations  of  both  approaches  include  a)  naive, 
overoptimistic  treatment  of  inevitable  uncertainties  in 
demands  for  support,  and  b)  simple  mechanistic  treatment  of 
repair  and  distribution  of  parts.  Further,  the  evaluation  of 
sortie  generation  capability  within  a  wide  spectrum  of 
resource  postures  in  a  combat  environment  has  been  difficult 
without  a  massive  simulation  effort. 

The  simulation  approach  to  ABO  issues  has  centered  on 
the  TSAR  (Theater  Simulation  of  Airbase  Resources)  model 
(Emerson  1982)  and  the  TSARINA  (TSAR  Inputs  using  AIDA)  model 
(Emerson  1980) ,  both  developed  by  the  RAND  Corporation  for 
the  U.S.  Air  Force.  TSAR  is  a  sortie  generation  model  while 
TSARINA  is  an  air  base  damage  model. 

TSAR  was  created  with  the  interdependencies  of  air  base 
resources  as  a  focal  point.  The  intent  was  to  permit 
decisionmakers  to  explore  the  air  base  as  a  system  in  order 
to  seek  improvements  to  that  system.  Resource  levels, 
logistics  policies,  environmental  factors  such  as  attrition, 
and  operational  tasking  can  all  be  varied  while  assessing  the 
impact  on  sortie  performance.  Weak  or  deficient  areas  in  the 
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logistics  infrastructure  can  thus  be  identified.  A  single 
air  base  can  be  simulated  or  an  entire  system  of 
interdependent  air  bases  can  be  modeled.  The  simulation  also 
allows  the  air  base  to  be  attacked  and  the  results  of  those 
attacks  examined.  Thus,  applications  of  the  model  are  broad 
and  encompass  a  wide  range  of  interest  by  many  different 
users . 

The  TSAR  model  is  a  discrete-event  Monte  Carlo 
simulation  which  models  11  classes  of  resources.  TSARINA  is 
a  companion  model  supporting  the  TSAR  sortie  generation 
simulation  program.  It  calculates  resource  damage  caused  by 
ettacks  on  the  air  base  as  well  as  toxic  effects  ~f  chemical 
attacks.  Many  types  of  weapons  can  be  simulated  with 
uncertainties  and  randomness  included.  Figure  1.1  summarizes 
the  relationship  of  the  models  and  their  primary  features. 
Complete  descriptions  of  both  models  and  capabilities  are 
found  in  Emerson  and  Wegner  (1S85) . 

The  goal  of  this  work  is  to  gain  more  complete  knowledge 
of  an  air  base  logistics  infrastructure,  through  the 
TSAR/TSARINA  simulation  models,  to  extend  the  benefits  of 
analytic  models  which  are  more  adept  at  assessing  large-scale 
systems.  Certain  objectives  will  guide  us  to  this  end. 

Research  Objectives 

A  major  focus  of  this  research  is  the  use  of  simulation, 
through  efficient  experimental  design,  to  identify  and 
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TSAR/TSARINA  for  Analyzing  Sortie  Generation  Capability 
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estimate  parameters  which  can  be  used  in  analytic  models.  As 
discussed  above,  both  simulation  and  analytic  approaches  have 
been  followed  in  addressing  the  issues  of  ABO  and  logistics 
infrastructure.  These  approaches  do  not  have  to  be 
exclusive;  rather  they  can  be  complementary.  The  better  and 
more  efficiently  we  use  large-scale  detailed  simulation 
models,  the  better  our  analytical  models  will  be. 

One  of  the  purposes  of  a  simulation  is  to  gain  insights 
and  understanding  of  a  real-world  system.  Although  the 
simulation  model  is  a  simpler  representation  of  reality,  it 
can  be  very  complex  in  and  of  itself.  Thus  an  even  simpler 
model  may  be  used  to  better  understand  the  complex  model; 
this  simpler, auxiliary  model  is  often  called  a  metamodel . 
Figure  1.2  depicts  the  relationship  between  the  real-world, 
the  simulation  model,  and  the  metamodel  (Friedman  1984)  . 
Friedman  (1984)  provides  numerous  references  for  discussions 
of  the  relationships  of  analytic  and  simulation  models,  and 
the  use  of  metamodels . 

This  research  proposes  to  bridge  the  gap  between 
simulation  and  analytic  models  by  capturing  the  detail 
provided  by  a  simulation  model  in  a  simpler  metamodel  which 
can  then  be  used  analytically  to  address  capability 
questions  .  This  provides  a  useful  and  analytically  sound 
tool  for  decision-makers  confronting  ABO  issues.  Figure  1.3 
captures  the  essence  of  the  problem  where  simulation  models 
are  used  to  generate  performance  data  based  on  the  level  of 
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ABDR  -  Aircraft  Battle  Damage  Repair 
AIS  -  Avionics  Intermediate  Shops 
RRR-  Rapid  Runway  Repair 


Figure  1.3 

Air  Base  Operability  Framework 
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policy/resource  variables  and  external  variables.  These  data 
then  will  be  used  to  estimate  simpler  metamodels  that  will 
become  the  forecast  generating  system.  Once  developed,  the 
metamodels  will  be  used  with  actual  inputs  of  resource  levels 
from  the  decision  maker's  system. 

The  specific  research  objectives  can  be  stated  as 
follows : 

1)  Efficiently  apply  an  experimental  design  that  will 
reduce  variance  due  to  the  inherent  randomness  of 
the  TSAR  and  TSARINA  simulation  models; 

2)  Estimate  metamodels,  with  significant  main  effects 
and  two-way  interactions,  from  large-scale 
simulation  experiments  so  that  sorties  flown  can  be 
predicted  based  on  input  factors; 

3)  Evaluate  the  impact  of  air  base  attacks  on  the 
number  of  daily  sorties  flown;  and 

4)  Identify  key  resources  and/or  interactions  ever  a 
thirty-day  time  period  with  and  without  air  base 
attacks . 


Qraaai ration  of  The  Thesis 
Chapter  II  describes  the  sortie  generation  process 
underlying  the  problem  area  and  provides  some  background  on 
various  approaches  to  modeling  sortie  generation  at  an  air 
base.  Also  we  introduce  some  background  on  statistically 
controlled  experiments  where  variance  reduction  is  a  concern. 
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Chapter  III  discusses  the  methodology  involved  in 
setting  up  the  TSAR/TSARINA  data  bases  and  describes  the 
metamodels  to  be  estimated.  Also  we  discuss  the  experimental 
design  and  how  each  research  objective  is  to  be  measured. 

Chapter  IV  centers  on  how  well  the  design  worked  with 
respect  to  variance  reduction  and  the  estimated  metamodels. 

It  concludes  with  an  evaluation  of  the  impact  of  attacks  on 
daily  sortie  counts. 

Chapter  V  focuses  specifically  on  the  interpretation  of 
the  estimated  metamodels.  It  concludes  with  an  overall 
evaluation  of  the  input  factors  in  the  case  where  the  air 
base  is  attacked  and  the  case  where  there  are  no  attacks. 

Chapter  VI  provides  overall  conclusions  for  this  thesis. 
First,  the  scope  of  the  research  is  discussed  to  emphasize 
the  appropriate  extendability  of  the  results.  Next, 
conclusions  are  presented  for  each  research  objective. 
Finally,  limitations  and  corresponding  future  research  are 
discussed . 
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CHAPTER  II  -  BACKGROUND  AND  LITERATURE  REVIEW 

Necessary  background  for  this  research  is  an 
understanding  of  the  sortie  generation  process,  i.e.,  what 
resources,  factors  and  processes  are  involved  in  flying 
sorties.  This  is  presented  first,  followed  by  discussions  of 
simulation  and  analytic  tools  used  to  model  sortie  generation 
of  today's  fighter  aircraft.  Next  is  a  review  and  critique 
of  a  previous  study  which  provides  the  backdrop  for  this 
work.  Lastly  there  is  a  section  on  the  underlying  theory  of 
statistically  controlled  experimental  designs. 

The_ Sortie  Generation  Process 
The  sortie  generation  process  is  complex  with  much 
inherent  variability.  Figure  2.1  captures  the  relationships 
of  important  factors  in  the  sortie  generation  process  at  an 
air  base.  Figure  2.2  further  extends  the  sortie  generation 
process  by  including  the  dynamics  of  the  resource 
interactions  in  an  environment  where  the  air  base  is 
vulnerable  to  attacks .  Interdependencies  among  factors 
W1^hin  the  logistics  infrastructure  are  evident  in  Figure 
2.2.  As  a  result  we  require  a  total  systems  perspective  over 
time  rather  than  simple  or.e-f actor-at-a-t ime  analyses.  The 
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Sortie  Generation  Process 


Airbase  System 
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Sortie  Generation  Dynamics 
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interdependencies  are  not  always  easy  or  obvious  to  explain. 
For  example,  if  we  envision  Figure  2.2  with  several  aircraft 
over  many  days,  it  is  possible  for  a  high  level  of  flying  in 
the  first  few  days  to  lead  to  aircraft  losses  which  result  in 
flying  fewer  sorties  later  than  otherwise  would  have  been 
flown . 

With  sorties  flown  (or  some  function  of  this  performance 
measure)  as  the  response  variable,  we  must  deal  with  the 
prospect  of  highly  correlated  responses  over  time.  From  one 
perspective,  we  expect  correlations  on  a  day-to-day  basis. 

In  the  absence  of  interventions,  such  as  attacks  on  the  base, 
the  leve_  of  flying  is  logically  related  to  the  health  of  the 
logistics  infrastructure.  Thus  it  seems  that  if  many  (few) 
sorties  are  flown  on  a  particular  day,  we  would  expect  to 
also  fly  at  the  same  level  the  next  day.  An  exception  is 
when  heavy  flying  leads  to  large  maintenance  backlogs;  many 
aircraft  must  be  repaired  and  the  flying  drastically  declines 
until  the  repair  workload  is  reduced.  This  situation  is 
generally  short-lived  and  high  levels  of  flying  can  resume 
following  the  recovery. 

Another  perspective  considers  that,  given  a  wartime 
scenario,  deliveries  of  resources  will  be  extremely  limited 
and  basic  logistics  capabilities  will  not  be  enhanced  within 
the  first  30  days.  As  a  result,  future  responses  are 
necessarily  related  to  that  beginning  posture  on  Day  1. 
Attacks  on  the  air  base  further  complicate  the  matter  by 
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creating  nonstationarities  and  interruptions  in  the  time 
series  data.  Here  we  expect  the  attacks  to  cause  large 
initial  degradations  in  performance  which  recover  to  some  new 
level  that  is  substantially  less  than  the  initial  capability. 

Sortie  generation  analysis  problems  arise  from  basically 
two  sources:  the  nature  of  the  data  and  the  modeling  of  the 
sortie  generation  process.  The  simulation  of  the  sortie 
generation  process  also  involves  certain  considerations  since 
it  is  a  simpler  representation  of  the  real  system.  A  major 
concern  is  the  amount  of  variability  inherent  in  the  problem 
itself  and  how  the  modeling  accounts  for  it.  For  example, 
how  sensitive  are  the  results  to  changes  in  the  random  number 
streams  used  by  the  model?  How  much  variability  is  due  to 
the  nature  of  the  process  and  how  much  to  the  model  itself? 

An  additional  analysis  issue  is  the  possibility  of 
nonconstant  variance  across  design  points  when  attacks  are 
included.  Folkeson  (1986)  indicates  that  this  may  be  the 
case.  Finally  the  problem  is  complicated  since  the 
simulation  measures  the  resource  levels  only  on  Day  1.  Thus 
we  are  dealing  with  indicator  variables  representing 
logistics  policies,  repair  capabilities,  expected  delivery 
schedules,  and  resource  levels  as  of  Day  1,  rather  than 
trackable  quantitative  series  for  the  independent  variables. 

Simulation  and  Analytic  Tools  for  Modeling  Air  Bases 

The  TSAR  and  TSARINA  models  are  prevalent  in  the 
detailed  study  of  air  bases  via  simulation.  These  models  are 
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described  above.  Additionally,  other  simulation  and  analytic 
models  have  been  developed  which  are  described  briefly  below. 
We  also  provide  a  summary  of  current  ABO  research  being  done 
with  simulation  tools . 

Simulation  Models 

Sortie  generation  models  other  than  TSAR  have  been 
developed  for  various  reasons,  but  usually  to  answer  specific 
questions  or  investigate  particular  areas  of  interest.  As  a 
result,  none  to  date  provide  the  detailed  relationships 
modeled  by  TSAR/TSARINA  which  give  it  flexibility  and  overall 
applicability  to  many  interests. 

A  less  complex  simulation  was  applied  to  a  multilevel 
maintenance  system  to  discover  design  parameters  which 
optimize  system  performance  (Chrissis  and  Gecan  1986) .  While 
a  novel  application,  it  is  a  simplistic  and  narrow  functional 
view  limited  to  the  supply  and  repair  of  spare  parts.  Also, 
as  the  authors  note,  their  approach  is  best  applied  to  small 
systems  since  dimensionality  is  a  serious  problem  as  the 
size  of  the  system  increases.  What  is  missing  as  a  result  is 
the  invaluable  insights  into  the  complex  interactions  of  the 
logistics  system  on  the  air  base. 

Hughes  Aircraft  has  developed  a  model  to  evaluate 
various  basing  and  logistics  options  for  aircraft  they  are 
designing  (Tinley,  1988) .  The  intended  use  appears  to  be  for 
the  evaluation  of  rather  broad  issues  such  as  levels  of 
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maintenance,  main  operating  bases  versus  dispersed  locations, 
spare  part  stockage  levels,  etc.  Again,  it  misses  the 
interactions  and  complexity  of  the  logistics  infrastructure. 

While  these  efforts  provide  valuable  insights  to  limited 
and  specific  questions,  analysts  continue  to  demand 
simulation  which  captures  many  functional  areas 
simultaneously  such  as  provided  by  TSAR/TSARINA. 

A  project  recently  undertaken  by  the  Air  Force  includes 
an  analytic  treatment  of  the  logistics  processes  within  a 
simulation  framework.  The  resulting  model  is  called  the 
Combat  Base  Assessment  Model  or  CBAM  (Hume,  1988) .  The 
purpose  of  CBAM  is  to  provide  a  single,  PC-based,  easy-to-use 
tool  to  simulate  attack,  repair,  and  sortie  generation  at  an 
air  base.  The  developers  describe  the  model  as  a 
"deterministic  repeatable  computational  model"  (Garjak, 

1988)  .  Since  CBAM  is  intended  to  be  run  on  a  desktop 
personal  computer,  multi-trial  results  are  not  practical. 
Therefore  CBAM  simplifies  the  use  of  random  variables  by  (a) 
replacing  the  random  variable  with  a  deterministic  value 
based  on  the  most  likely  value,  (b)  truncating  distribution 
tails,  (c)  using  selection  criteria  to  limit  choices,  (d) 
stratifying  outcomes  from  continuous  distributions  into 
discrete  states,  and  (e)  compressing  distributions  to  make 
them  more  "spiky"  (Garjak,  1988)  .  This  most-likely  approach 
to  simulation  allows  simplification,  but  also  tends  to  fail 
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in  capturing  the  daily  uncertainty  and  volatility  within  the 
logistics  infrastructure  in  a  wartime  environment. 

An  Analytic  Model 

Analytic  approaches  have  centered  on  the  Dyna-METRIC 
(Dynamic  Multi-Echelon  Technique  for  Recoverable  Item 
Control)  model  also  developed  by  RAND  (Hillestad  1982)  .  The 
focus  in  these  efforts  is  on  the  stockage  and  movement  of 
reparable  spare  parts  through  the  various  levels  of 
controlled  inventory  and  maintenance. 

The  overall  purpose  of  Dyna-METRIC  is  to  relate  aircraft 
spare  parts  inventory  levels  and  maintenance  capability  to 
the  readiness  of  aircraft.  As  described  by  Hillestad  (1982), 
a  key  characteristic  of  the  mathematical  model  is  the  ability 
to  deal  with  dynamic  or  transient  demands  for  spare  part 
inventories  and  component  repair  caused  by  a  changing 
environment.  The  model  implements  a  set  of  analytic 
equations  which  capture  the  dynamic  behavior  of  the  component 
repair  queueing  system.  From  these  equations,  time-dependent 
inventories  can  be  computed  and  related  to  aircraft 
capabilities  to  fly  assigned  missions. 

Current  A3Q  Research 

Current  ABO  research  efforts  primarily  use  simulation  as 
an  analysis  tool.  For  example,  the  Air  Force  is  developing 
ABO  reference  manuals  which  describe  the  operational 
capability  gained  by  adding  certain  ABO  assets  to 
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representative  bases  (Hume,  1988).  Based  on  TSAR 
simulations,  these  manuals  will  provide  information  for 
logistics  planners. 

Another  TSAR-based  study  analyzes  various  ABO  policies 
and  attempts  to  capture  the  sensitivities  of  the  dynamic 
wartime  environment  (Folkeson,  1988) .  A  major  focus  in  this 
effort  is  the  variance  of  attack  effects.  Fifty  different 
at  cks  are  used  when  evaluating  each  policy.  The  emphasis 
is  not  so  much  on  the  detailed  performance  of  the  logistics 
infrastructure,  but  on  the  performance  and  survival  of  the 
air  base  system.  Hence  the  focus  is  more  on  the  "defend"  and 
"survive"  aspects  of  ABO  rather  than  the  "recover"  and  "fly" 
tasks  that  we  are  concentrating  on  in  this  paper. 

Other  work  has  used  TSAR  as  a  central  tool  for  assessing 
the  capability  of  an  air  base  (Manger,  1988)  .  A  prototype 
model  has  been  developed  which  consists  of  pre-  and  post¬ 
simulation  data  processors  built  around  TSAR.  These  data 
processors  use  standard  data  systems  to  create  TSAR  databases 
for  actual  air  bases  using  current  data.  The  model  produces 
sortie  capability  information  and  predicts  problem  resource 
items.  Such  a  model  could  be  used  as  an  on-line  real-world 
capability  assessment  tool. 

A  Previous  Study  of  an  A?  r  Base  Logistics  Infrastructure 

One  study  in  particular  addresses  the  overall  air  base 
resourcing  problem  from  a  total  systems  perspective. 
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Folkeson  et  al.  (1986)  used  TSAR  to  simulate  the  air  base 
logistics  support  system  defined  by  nine  resources  which 
contribute  to  flying  an  aircraft.  Figure  2.1  shows  how  these 
variables  relate  to  the  sortie  generation  process. 

The  .LQ-qIs tics  Variables 

Each  of  the  nine  variables  was  varied  over  three  levels: 
high,  medium,  and  low.  This  resource  structure  was  subjected 
to  a  constant  environment,  i.e.,  the  environment  was  not 
changed  as  the  resource  structure  was  varied.  This 
environment  included  such  factors  as  aircraft  attrition 
(i.e.,  the  rate  at  which  aircraft  are  lost  and  do  not  return 
to  the  air  base) ;  a  fixed  number  of  attacks  on  the  air  base 
which  determine  when  and  how  many  resources  are  lost;  and 
sortie  demands  placed  on  the  air  base  (i.e.,  how  many  sorties 
the  air  base  unit  tries  to  fly) . 

The  Experimental  Design  and  Model 

Sortie  generation  ar.d  air  base  attack  results  were 
determined  using  the  TSAR  and  TSARINA  models.  The  experiment 
was  based  on  a  fractional  factorial  design  by  Box  and  Behnken 
(1960)  with  130  simulation  runs  required  versus  19,683  for 
the  full  factorial.  Twenty  replications  were  also  made  for  a 
total  of  2600  simulation  runs.  Although  the  3ox  and  Behnken 
design  was  to  estimate  a  second  degree  graduating  polynomial, 
the  assumed  model  was  stated  as: 


where 
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’i  "  V1'  *  +  Si 

1  =  1  J  J 


=  sorties  flown  on  Day  i 
=  level  of  resource  j  on  Day  1 
Bj(i)  =  beta  coefficient  for  3th 

resource  on  Day  i 
Regression  was  used  to  find  a  system  of  equations,  one 
for  each  day,  to  predict  the  effect  of  changes  in  variable 
level . 

Findings  and  Conclusions 

One  conclusion  of  the  Folkeson  (1986)  study  is  that 
different  factors  or  resources  appear  to  be  important  at 
different  times.  Thus  the  resourcing  decision  depends  on  how 
long  one  expects  the  "war"  to  last.  If  one  plans  on  14  days, 
but  then  must  operate  over  30  days,  the  results  may  not  be 
favorable  because  the  "wrong"  resources  were  invested  in. 

For  example,  personnel  seemed  to  have  its  biggest  effect  in 
the  early  days,  while  fuel  (POL)  contributed  to  sorties  later 
in  the  war.  Ideally  these  contributions  must  be  balanced  to 
ensure  the  highest  level  of  sortie  generation  possible  within 
budget  constraints. 

Problems  With  the  Design 


To  estimate  a  model  with  only  main  effects,  as  was  done, 
would  require  far  fewer  runs,  and  a  two-level  design,  rather 
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than  three-level,  could  have  been  used.  The  same  results 
should  be  obtained  with  only  32  runs  (a  1/16  fractional 

9 

replication  of  a  2  full  factorial  experiment).  For  the 
number  of  simulation  runs  and  computer  time  involved  in  this 
experimental  design,  the  inference  space  is  extremely 
limited;  they  can  only  suggest  results  for  a  single  air  base 
under  a  single,  very  specific  scenario.  The  experiment  does 
not  allow  for  an  estimate  of  the  impact  of  the  attacks  on 
sortie  generation.  It  also  makes  no  account  for  the 
randomness  found  in  the  TSARINA  attacks;  blocking  is  used  to 
account  for  the  randomness  only  in  TSAR,  but  then  is  not 
included  in  the  regression,  thus  ignoring  any  variance 
reduction  gained  by  blocking.  Finally,  as  recognized  by  the 
authors,  interaction  terms  may  be  very  significant  and  need 
to  be  included  in  the  estimated  model . 

St  at  i  s,t  i c a  1  lx =  Con t  r,pi  lad. . Exp.ar.imeflt.al  -D.e.si  ga 
Much  has  been  written  concerning  the  design  of 
experiments  in  order  to  achieve  certain  statistical 
characteristics.  Steinberg  and  Hunter  (1984)  provide  a 
review  of  this  research  area  and  include  an  extensive 
bibliography.  We  focus  on  a  specific  issue  related  to 
experimental  design  which  is  the  development  and  use  of 
variance  reduction  techniques  (VRT) . 
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Variance  Reduction  Techniques  In . Simulation 

As  discussed  by  Wilson  (1984),  the  primary  disadvantage 
of  using  simulation  models,  particularly  those  that  model 
large-scale  systems,  to  generate  experimental  data  is  the 
large  number  of  often  lengthy  simulation  runs  required  and 
the  associated  computing  cost.  Because  of  these  high  costs, 
various  VRTs  have  been  developed  which  attempt  to  reduce  the 
number  of  runs  needed  to  achieve  the  desired  level  of 
precision  for  the  parameters  to  be  estimated. 

In  general,  simulation  experimenters  run  their  model 
some  number  of  times  to  generate  observations  of  the  random 
variable  Y,  the  response  of  interest.  Following  Wilson 
(1984),  the  simulation  model  is  a  response  function  \[/(-)  with 
inputs  {IL:  i  £  1}  of  independent  random  numbers.  Assuming  a 
finite  upper  bound  m  on  the  number  of  input  variates  sampled 
during  one  run  of  the  model,  the  set  of  arguments  for  \|/  ( ■ ) 
can  be  represented  by  an  m  x  1  random  vector 

U  -  [V . UJ'.  Thus  Y  «  y(U)  and  6  =  E  (Y)  .  From  the 

sample,  the  sample  mean  Yn  is  computed  for  the  response  Y 
over  n  independent  replications  of  the  simulation  model.  Yn 
is  an  unbiased  estimator  of  E(Y)  =  8,  the  population  mean 
response,  with  Var(Yn)  =  Var(Y)/n.  The  desired  result  when 
using  a  VRT  is  to  estimate  0n  where  E(0n)  =  0  and  Var(0n)  < 
Var(Yn).  Wilson  (1984)  further  develops  this  discussion  by 
including  efficiency  measures  for  evaluating  VRTs. 
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Two  categories  of  VRTs  are  delineated  by  Wilson  (1984)  : 
correlation  methods  and  importance  methods.  In  this  research 
we  focus  on  correlation  methods  which  use  the  linear 
correlation  of  the  simulation  responses  to  improve  variance 
results.  Even  more  specifically,  we  can  focus  on  the  use  of 
random  numbers  which  are  integral  cogs  in  the  machinery  of 
Monte  Carlo  simulations. 

Random  Number  .As s igniter. t  in  Monte  Carlo.  Simulation 

How  the  random  numbers  are  used  in  a  simulation  can  be 
controlled  to  some  extent  by  the  experimenter.  A  single 
random  number  stream  may  be  generated  and  used  sequentially 
whenever  random  input  variates  are  needed  by  the  model. 
Another  approach  is  to  use  different  streams  for  individual 
components  or  processes  within  the  model,  thus  linking 
randomness  as  determined  by  the  random  number  stream  more 
directly  to  a  process.  Schruben  and  Margolin  (1978)  provide 
an  example  of  a  hospital  simulation  which  uses  a  set  of  six 
random  number  streams  where  each  stream  drives  a  particular 
stochastic  component  of  the  model.  Whichever  approach  is 
used,  a  random  number  stream  is  usually  generated  from  some 
starting  value  or  seed  which  allows  the  stream  to  be 
reproduced  and  controlled  by  the  experimenter.  Thus  the 
randomness  of  the  simulation  can  be  related  to  the  beginning 
seed(s)  as  determined  by  the  experimenter. 
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When  planning  the  simulation  experiment,  the  researcher 
must  determine  the  random  number  streams  to  be  used  to 
generate  each  design  point  in  the  experiment.  Schruben  and 
Margolin  (1978),  Wilson  (1984),  and  Brately  et  al.  (1983) 
provide  the  basis  for  the  following  brief  discussion  of  three 
assignment  alternatives.  Suppose  we  have  a  parameter  Y°  to 
be  estimated.  Let  Yi  =  \\fi  (U]_)  and  Y2  =  V2  (u2)  •  There  are 

three  basic  methods  to  approach  the  simulation  experiment 
with  respect  to  the  use  of  the  random  number  streams. 

Independent  streams  is  a  method  where  different, 
randomly  chosen  streams  are  used  at  each  design  point. 
Typically,  this  approach  results  in  uncorrelated  samples. 

Here  we  simply  make  independent  runs  to  obtain  a  single 
observation  of  Y° . 

Common  random  numbers  is  the  approach  where  the  same  set 
of  random  number  streams  is  used  at  two  or  more  design 
points.  Positive  correlations  between  the  design  points  are 
typically  induced.  Thus,  if  we  are  interested  in  estimating 
the  difference  E(Yi)  -  E(Y2),  this  method  could  be  used  to 
reduce  the  variance  of  Y°  =  Yi  -  Y2 

Var(Yi  -  Y2)  =  Var  (Yx)  +  Var(Y2)  -  2  Cov(Yi,Y2) 
by  inducing  Cov(Yi,Y2)  >  0. 

Antithetic  variates  is  a  third  assignment  method  where  a 
seed  vector  is  used  at  one  design  point,  and  then  each  random 
number  is  subtracted  from  unity,  creating  a  "new"  stream 
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which  is  used  at  a  second  design  point.  Thus  if  random 
number  stream  R  =  (r^,  ...),  then  the  antithetic  stream 

R  =  (1  -  r^,  1  -  r^t  ...) .  This  method  tends  to  induce  a 

negative  correlation  between  the  samples.  For  example, 
suppose  we  wanted  to  estimate  E(Yi)  where  Yi,  Y2  represent 
replicates  so  that  \j/]_  =  \|/'2  -  This  method  could  be  used  to 
reduce  the  variance  of  Y°  =  ~(Yi  +  Y2) 

Var  [j(Yi+Y2)  ]  =jVar(Y!)  +^-Var(Y2)  +  jCov(Yi,Y2) 

«  \  Var  (Yi)  +  Cov(Yi,  Y2) 
by  inducing  Cov(Yi,Y2)  <  0. 

We  should  note  also  that  these  techniques  of  random 
number  assignment  do  not  guarantee  variance  reduction. 
Successful  application  depends  on  the  structure  of  the  model 
and  the  particular  problem  (Wilson,  1984) . 
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CHAPTER  III  -  METHODOLOGY 


An  F-15  aircraft  data  base  currently  being  used  by  the 
RAND  Corporation  and  various  Air  Force  agencies  is  simulated 
using  the  TSAR/TSARINA  models.  The  goal  is  to  identify  key 
resource  factors  and  significant  two-way  interactions  and 
explore  the  nature  of  the  relationship  between  the 
independent  variables  and  the  response  variable.  First  we 
discuss  the  factors  to  be  modeled;  each  is  described  along 
with  its  relationship  to  sortie  generation  and  the  resource 
levels  used  in  the  experiment.  Next  we  examine  the 
metamodels  to  be  estimated  and  the  assumptions  of  each.  Then 
the  experimental  design  is  explained.  The  final  section 
discusses  how  the  research  objectives  listed  in  Chapter  I 
will  be  assessed. 

The  Resource  Factors 

Nine  resource  factors  are  included  as  variables  in  this 
research.  These  were  selected  based  on  previous  research  and 
experimentation  with  the  TSAR  model  and  have  shown  to  be 
important  factors  in  the  sortie  generation  process.  Figure 
3.1  depicts  the  relationship  of  these  factors  to  ouhers  in 
the  TSAR  and  TSARINA  F-15  data  bases.  Since  a  two-level 
experimental  design  is  used,  each  factor  is  evaluated  at  a 


TSARINA 


TSAR/TSARINA  Database  Factors 
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"high",  or  more  favorable,  level  and  a  "low",  or  less 
favorable,  level. 

To  be  realistic  and  useful,  the  derived  metamodel  must 
capture  a  wide  spectrum  of  various  logistics  resource 
positions  as  well  as  elements  of  an  uncertain  wartime 
environment  such  as  air  base  attack  and  attrition.  Thus  the 
high  and  low  levels  for  each  variable  are  chosen  so  that  we 
have  a  valid  and  realistic  inference  space.  The  high  levels 
represent  the  logistics  infrastructure  one  would  expect  to 
find  supporting  72  F-15  aircraft.  In  an  era  of  tight  defense 
budgets,  it  is  very  unlikely  we  would  find  one  with  greater 
resources.  Given  the  "high",  it  seems  logical  to  then 
degrade  it  to  develop  the  "low"  level  for  each  resource.  The 
inference  space  thus  enclosed  should  be  realistic  for  most 
situations  encountered  by  decision  makers  when  forecasting 
with  a  simpler,  derived  metamodel. 

Each  factor  also  represents  an  actual  functional  area 
found  within  the  logistics  infrastructure  of  a  tactical  air 
base.  Each  factor  is  described  below  regarding  its  high 
level,  low  level,  and  general  impact  on  sortie  generation. 
Tables  3.1  and  3.2  below  summarize  the  high  and  low  levels  of 
the  resources  respectively. 

r.actQl  3; - Filler  or  Replacement  Aircraft 

Seventy-two  fighter  aircraft  are  normally  assigned  to  a 
tactical  air  base  with  three  squadrons.  Each  of  these  flying 
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High  Levels 
FACTOR 

B  -  Aircraft  = 

C  -  ABDR  Capability  = 

D  -  Recovery  = 

E  -  Personnel  = 

F  -  AIS 

G  -  Support  Equip  = 

H  -  Spares  = 

J  -  Missiles  = 

K  -  Fuel 


Table  3.1 

for  the  Resource  Factors 
HIGH  LEVEL 

72  assigned  plus  18  filler  aircraft 
available  with  72  hour  delay 

6  assessors  (2  per  AMU)  where  work 
cannot  begin  until  damage  is 
inspected  by  a  trained  assessor 

Full  range  of  improved  procedures 

which  includes  manual  workarounds; 
CE  and  EOD  personnel  and  equipment 
also  included 

Typical  quantities  expected  —  assume 
these  are  the  number  authorized  by 
specialty 

2  sets  with  5  stations  each 

Typical  quantities  expected  —  assume 
these  are  number  authorized 

Computed  by  TSAR  with  100%  safety 
factor  using  AFM  67-1  Chap  11 
procedures 

Initial  Stocks  -  300  AIM9-M 

300  AIM7-M 

612  AIM9-M  components 
424  AIM7-M  components 

Day  1  -  delivery  of  AIM-9Ms  &  AIM-7MS 

Days  2,5,10,15  -  deliveries  of  AIM-9M 
and  AIM-7M  components 

Deliveries  arrive  Days  10,15,20,25 
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Table  3.2 


Low  Levels  for  the  Resource  Factors 


FACTOR 


-  Aircraft 


LOW  LEVEL  (A  DEGRADATION  OF  HIGH  CASE) 

=  72  assigned  with  no  filler  aircraft 

=  available 


ABDR  Capability  =  3  assessors  (1  per  AMU)  where  work 

=  cannot  begin  until  damage  is 
=  inspected  by  a  trained  assessor 


-  Recovery 


-  Personnel 


=  Slower  alternate  procedures; 

=  CE  and  EOD  personnel  and  equipment 
=  reduced  to  75%  of  high  level 

=  Quantities  reduced  to  75%  of  high 
=  level 


AIS  =  l  set  with  5  stations 

Support  Equip  =  Quantities  reduced  to  75%  of  high 

=  level 


-  Spares 


-  Missiles 


-  Fuex 


=  Computed  by  TSAR  with  10%  safety 
=  factor  using  AFM  67-1  Chap  11 

*  procedures 

=  Initial  Stocks  -  same  as  high  case 

=  Days  5,10,15  -  deliveries  of  AIM-9M 
=  and  AIM-7M  components 

=  Deliveries  arrive  Days  10  and  20 


squadrons  is  supported  by  an  Aircraft  Maintenance  Unit  or  AMU 
in  addition  to  centralized  repair  shops.  During  wartime, 
filler  or  replacement  aircraft  may  be  available  to  offset  the 
loss  of  aircraft  due  to  attrition  and  air  base  attacks.  As  a 
general  rule,  the  logistics  infrastructure  is  designed  to 
support  no  more  than  seventy-two  aircraft  at  any  time.  When 
a  loss  occurs,  a  replacement  is  requested.  If  available,  the 
filler  arrives  after  a  period  of  delay. 

High  Level:  Seventy-two  aircraft  are  assigned  to  the 
base  with  an  additional  18  filler  or  replacement  aircraft 
available,  each  with  a  72-hour  delay. 

Low  Level :  Seventy-two  aircraft  are  assigned,  but  no 
filler  aircraft  are  available. 

Impact :  More  aircraft  available  increases  the  potential 

to  fly  more  sorties  by  keeping  the  air  base  closer  to  its 
full  complement  of  aircraft. 

Factor  C;  Aircraft  Battle  Damage  Repair  (ABDR1  Capability 
ABDR  is  the  specialized  repair  of  aircraft  that  return 
to  the  air  base  with  damage  caused  by  enemy  ground  artillery, 
antiaircraft  weapons,  or  air-to-air  combat.  The  nature  of 
the  damage  requires  the  use  of  special  techniques  ar.d 
materials.  Highly  trained  assessors  must  plan  and  direct  the 
repair  actions.  Thus,  the  number  of  assessors  limits  the 


amount  of  A3DR  that  can  be  accomplished  at  any  given  time. 
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High  Level:  Six  assessors  (2  per  AMU)  are  available. 
Repair  work  cannot  begin  until  a  damaged  aircraft  has  been 
inspected  by  a  trained  assessor. 

Low  Level:  Only  three  assessors  (1  per  AMU)  are 

available  . 

Impact :  An  increase  in  the  number  of  assessors 

decreases  the  number  of  battle-damaged  aircraft  waiting  to 
begin  repairs.  As  a  result,  ABDR  capability  increases,  and 
more  aircraft  are  available  sooner  for  flying  sorties. 

Factor  D;  Recovery  From  Air  Base  Attack 

"Recovery"  centers  on  the  repair  of  damaged  runways  and 
taxiways  so  that  flying  can  resume.  EOD  or  Explosive 
Ordnance  Disposal  personnel  must  first  clear  unexploded 
munitions  dropped  by  the  attackers  from  areas  in  the  vicinity 
of  damaged  pavement.  Then  holes  in  the  pavement  must  be 
filled,  rubble  removed,  and  pavement  poured  by  Civil 
Engineering  (CE)  personnel.  When  a  minimum  size  take-off  and 
landing  surface  is  available  and  accessible  to  the  aircraft, 
flying  operations  can  resume. 

Hioh  Level:  A  full  range  of  procedures  is  simulated 
including  manual  workarounds.  Also  included  are  full 
authorizations  of  civil  engineering  and  ordnance  disposal 
personnel  and  equipment  necessary  to  support  the  size  of  the 
air  base. 
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)  Low  Level:  Slower  alternate  procedures  are  used  instead 

of  the  normal  methods.  CE  and  EOD  personnel  and  equipment 
levels  are  also  reduced  to  75%  of  the  "high"  level. 

Impact :  Higher  recovery  capability  results  in  runways 

and  taxiways  being  opened  sooner  and  thus  leads  to  more 
sorties . 

F3C.tar_.E.i _ Maintenance  Personnel 

Maintenance  technicians  typically  are  trained  to  repair 
and  maintain  specific  aircraft  systems.  For  example,  there 
are  jet  engine  specialists,  navigation  systems  specialists, 
etc.,  as  well  as  generalists  like  crew  chiefs  who  are 
responsible  for  the  overall  condition  and  servicing  of  the 
\  aircraft.  The  number  of  crew  chiefs  and  each  type  of 

f 

specialist  assigned  to  the  air  base  is  determined  by  the 
number  of  aircraft  to  be  supported.  Thus  certain  manpower 
levels  are  determined  and  "authorized"  to  support  seventy-two 
aircraft . 

Hlah  Level:  Here  we  have  the  "typical"  number  expected 
by  specialty.  This  accounts  for  the  fact  that  full 
authorizations  are  often  not  reached  due  to  personnel 
shortages  and  long  training  pipelines. 

Low  Level:  Quantities  of  each  specialist  are  reduced  to 
75%  of  the  "high"  level. 

Impact :  More  maintenance  people  allow  more  aircraft  to 
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be  worked  at  any  point  in  time,  thus  leading  to  quicker 
repairs  and  more  flying. 

Z3.CLQIL.  V  ; - Avionics  Intermediate  Test  Station  (AIR) 

The  avionics  on  the  F-15  aircraft  are  very  sophisticated 
and  require  special  diagnostic  equipment  (i.e.,  the  AIS)  for 
their  repair.  Ideally  the  malfunctioning  "black  boxes"  are 
removed  and  replaced  with  a  spare  or  extra  working  component . 
However,  at  some  point  the  broken  ones  must  be  repaired  and 
this  requires  the  AIS  equipment.  Further  complicating  the 
process  is  the  fact  that  the  AIS  itself  is  highly  susceptible 
to  failure  and  requires  repair  of  its  own  components. 

High  Leve.l, :  Two  AIS  sets,  each  consisting  of  5 
)  stations,  are  available  to  repair  the  avionics  "black  boxes." 

L-Qw.  .Level:  Only  one  set  with  5  stations  is  available. 

Impact. :  The  availability  of  two  sets  increases  the 
likelihood  that  avionics  components  can  be  repaired  and  made 
available  to  fix  aircraft  which  can  then  fly. 

Factor  G:  Support  Equipment 

Support  equipment  include  all  the  equipment  necessary  to 
repair  an  aircraft  and/or  prepare  it  for  flying.  Examples 
include  fuel  trucks,  tow  tractors,  power  carts,  missile 
^-ra^le-sf  maintenance  stands,  etc.  Like  personnel,  certain 
quantities  of  each  type  are  "authorized"  based  on  the  number 
of  aircraft  assigned  to  the  air  base. 
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High  Level:  Quantities  are  the  expected  number  of  each 
type  of  equipment  typically  available.  Due  to  budget 
shortages  and  procurement  pipelines,  full  authorization 
levels  are  often  not  realized. 

Low  Level:  Quantities  are  reduced  to  75%  of  "r igh" 

level . 

Impact :  When  more  equipment  is  available,  more  aircraft 

can  be  worked  on  at  any  point  in  time,  and  thus  the  repair 
time  is  shortened  and  more  aircraft  are  available  for  flying. 

Factor  H:  Spare  Parts 

Spare  parts  are  used  to  replace  damaged  or  defective 
parts  on  the  aircraft  and  its  systems,  support  equipment,  and 
the  AIS .  The  type  and  number  of  parts  stocked  typically 
depend  on  the  number  of  aircraft  assigned  and  the  expected 
number  of  sorties  to  be  flown. 

High  Level:  Stockage  levels  by  type  are  computed  by  the 

TSAR  model  with  a  100%  safety  factor  based  on  AFM  67-1 
Chapter  11  procedures. 

Low  Level:  Levels  are  computed  with  a  10%  safety 
factor . 

Impact :  With  spares  available,  defective  parts  can  be 

removed  and  replaced  with  good  parts  and  the  aircraft  is 
ready  to  fly.  The  defective  part  is  repaired  later  and 
becomes  a  spare.  However,  when  no  spare  is  available,  the 
defective  part  must  be  removed,  repaired  in  the  appropriate 
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shop,  and  then  returned  to  the  aircraft.  Thus,  more  spare 
parts  leads  to  faster  repairs  and  more  aircraft  available  for 
flying . 

_ Missiles 

Missiles  are  the  primary  munition  of  the  F-15;  without 
missiles  these  aircraft  cannot  fly  effective  missions.  Thus 
aircraft  will  not  take  off  without  missiles  in  this 
simulation.  Two  types  of  missiles  are  available  and  are 
assembled  from  component  parts  by  munitions  specialists. 

High  Level:  Initial  stocks  of  built-up  missiles  are 
available  as  well  as  component  parts  to  build  more. 

Additional  missiles  are  delivered  on  Day  1.  Additional 
components  are  delivered  on  Days  2,  5,  10,  and  15. 

Low  Level:  The  same  initial  stocks  as  in  the  "high" 
case  are  available.  The  delivery  schedule  is  also  the  same, 
but  no  deliveries  are  received  on  Days  1  and  2.  Here  we 
project  that  deliveries  may  not  arrive  due  to  enemy  attack 
and/or  sabotage. 

Impact :  More  missiles  available  means  more  sorties  can 

potentially  be  flown,  i.e.,  there  is  less  chance  that  a 
sortie  will  not  be  flown  because  of  a  missile  shortage. 

Factor  Y. :  Fuel 

Aircraft  must  be  refueled  before  each  flight.  Fuel  is 
delivered  and  pumped  by  a  special  vehicle  or  can  be  pumped 
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from  a  fixed  refueling  point.  Storage  tanks  and  delivery 
vehicles  are  susceptible  to  sabotage  and  air  base  attacks. 

High  Level:  Initial  stocks  are  on-hand  plus  deliveries 
arrive  on  Days  10,  15,  20,  and  25. 

Low  level:  The  same  initial  stocks  are  available  and 
the  delivery  schedule  is  the  same.  However,  no  deliveries 
arrive  on  Days  15  and  25. 

Impact :  Aircraft  cannot  fly  without  fuel.  More  fuel 
available  lessens  the  chance  of  a  sortie  not  flown  because  of 
fuel  shortages. 


The.  Environmental  Factors 

This  research  also  includes  two  environmental  factors, 
where  we  define  the  environment  to  be  the  hostile  arena  in 
which  sorties  are  to  be  flown.  Such  factors  are  not  within 
the  managerial  control  of  the  air  base  leaders  and  decision¬ 
makers.  Rather,  the  factors  must  be  reacted  to.  Here,  the 
environmental  factors  we  model  as  variables  are  attacks  upon 
the  air  base  and  the  attrition  inflicted  upon  our  own 
aircraft  while  flying  sorties.  A  third,  locally 
uncontrollable  factor  is  the  amount  of  flying  the  unit  is 
tasked  to  fly.  Each  factor  is  described  below  and  is  shown 
in  Figure  3.1  as  it  relates  to  the  other  factors  included  in 


the  TSAR  and  TSARINA  models. 
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Me  -Base  Attacks, 

Enemy  attacks  on  the  air  base  destroy  resources,  create 
damage  which  inhibits  or  stops  flying  operations,  and,  in 
general,  disrupt  normal  operations.  When  and  how  many  times 

an  air  base  will  be  attacked  is  unknown;  however,  due  to  the 

l 

effectiveness  of  attacks  in  disrupting  sortie  generation, 
they  must  be  expected.  Generally,  as  time  passes,  the 
enemy's  ability  to  deliver  an  attack  is  expected  to  decrease. 

We  use  the  TSARINA  model  to  simulate  attacks  on  the  air 
base.  Conventional  (i.e.,  nonnuclear  and  nonchemical) 
attacks  by  enemy  aircraft  are  assumed  which  focus  on  runways, 
taxiways,  and  aircraft  shelters.  This  type  of  attack  can  be 
quite  effective  in  preventing  sorties  from  being  flown.  The 
attacks  are  optimized  from  the  enemy's  perspective  with 
regard  to  aimpoints  and  time  of  attack.  Six  attacks  occur  in 
the  first  five  days  and  are  summarized  in  Table  3.3. 

At-fc  rifcian  o  f...Q.ux-Aix.c,r.alt 

Every  time  a  sortie  is  flown,  the  aircraft  is  exposed  to 
various  risks  which  may  result  in  its  loss.  Risks  include 
mechanical  malfunctions,  acts  of  nature,  enemy  ground  fire  or 
air-to-air  attack,  etc.  These  risks  are  expressed  as  a  loss 
rate  per  sortie  flown.  Attrition  factors  are  modeled  within 
TSAR  either  based  on  time  periods  or  on  the  number  of 
friendly  sorties  flown.  The  high  and  low  levels  are 


described  below. 


Summary  of  the  Air  Base  Attacks 
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Hioh  Level.  The  high  or  more  favorable  level  of 
attrition  is  modeled  as  a  stepwise  reduction  of  the  attrition 
rate  from  1.2%  to  1.0%  of  sorties  flown.  The  timing  of  the 
reduction  is  based  on  the  number  of  sorties  flown  by  our 
aircraft.  The  attrition  rate  starts  at  1.2%  and  drops  to 
1.1%  when  we  have  flown  a  total  of  166  sorties.  When  the 
cumulative  number  of  sorties  flown  reaches  327,  the  attrition 
rate  further  declines  to  1.0%  where  it  remains  for  the  rest 
of  the  30-day  period.  The  logic  behind  the  reduction  is  that 
the  more  we  fly,  the  less  effective  the  enemy  is  against  us, 
and  thus  the  attrition  rate  is  driven  down.  If  we  cannot  fly 
many  sorties,  the  enemy  maintains  a  higher  level  of 
effectiveness  against  our  aircraft.  Thus,  the  more  we  can 
fly,  the  faster  our  attrition  rate  declines. 

Low  Level.  The  low  or  less  favorable  level  of  attrition 
is  a  constant  1.2%  per  sortie.  Here  our  sorties  have  no 
effect  on  the  effectiveness  of  the  enemy  against  our 
aircraft . 

M_r  Tasking  Orders  (ATO) 

Each  air  base  is  tasked  to  fly  during  wartime  by  a 
central  agency.  These  taskings  will  generally  exceed  the 
rates  flown  in  peacetime.  In  this  research  we  have  assumed 
that  the  taskings  will  be  high  and  will,  in  general,  exceed 
the  capability  of  the  air  base.  This,  in  effect,  creates  a 
"fly  as  much  as  you  can"  scenario  where  we  can  focus  on  the 
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logistics  capabilities  of  the  air  base.  Over  time,  the  level 
of  tasking  decreases,  but  is  still  higher  than  peacetime 
rates.  This  factor  is  treated  as  a  constant  in  all 
simulation  runs. 


The_Met_amodels  to  be  Estimated 
The  goal  of  our  analyses  is  to  derive  a  useful  and 
realistic  metamodel  which  captures  the  key  main  effects  and 
any  significant  two-way  interactions  between  the  main 
factors.  Higher  order  interactions  are  assumed  to  be 
neglible.  Additionally  we  want  to  evaluate  the  dynamic 
influences  of  significant  factors  and  interactions  over  time, 
with  and  without  attacks  on  the  air  base.  We  will  first 
discuss  the  metamodel  for  the  case  where  there  are  no  attacks 
and  then  the  case  where  attacks  are  present. 

The  MetflinQ-daX-  £sr  the  £ase.  af  No  At.tacisa 

The  metamodel  for  each  day's  flying  performance  is 
assumed  to  be  linear  and  comprised  of  the  nine  main  resource 
effects  plus  the  environmental  variable  (i.e.,  our  attrition) 
as  well  as  two-way  interaction  terms.  The  daily  model  is  of 
the  following  form: 


10 

Si  =  PoU)  +  I  Pj(i)Xij 

j-1 

9  10 

+  I  I  Pjk(i)XijXik  +  £(i)  (3.1) 

j-1  k-j+1 


i 
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where  =  the  number  of  sorties  flown  on  Day  i 
=  level  of  factor  j  on  Day  1,  and 

£(i)  reflects  the  variability  due  to  the  random¬ 
ness  within  the  logistics  processes  and  attrition 
in  addition  to  the  experimental  error,  where  £(i) 

~  N  (0,C£2)  . 

The  Attack  Case  Metamodel 

When  the  air  base  is  attacked,  we  have  in  effect  an 
eleventh  factor  which  influences  the  sortie  generation 
process.  The  model  to  be  estimated  is  the  same  except  now 
the  error  term  contains  variability  due  not  only  to  the 
logistics  processes  and  attrition,  but  also  the  variability 
due  to  the  attacks.  The  model  is: 

*  *  10  *  9  10 

Si  -  P0(i)  +  I  MDXij  +  I  £ 

j=l  J  j-l  k-j+1 

P*k(i)XijXlk  +  e*(i)  (3.2) 

* 

where  =  the  number  of  sorties  flown  on  Day  i 

X^j  =  level  of  factor  j  on  Day  1,  and 
★ 

£  (i)  reflects  the  variability  due  to  the 

randomness  within  the  logistics  processes  and 

attrition  plus  the  variability  due  to  the  air  base 

attacks  in  addition  to  the  experimental  error, 

where  £* (i)  ~  N (0,0  *2)  . 

£ 
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Although  the  form  of  the  metamodel  is  the  same  for  both 

cases,  the  beta  coefficients  are  not  assumed  to  be  the  same. 

★ 

In  fact,  imbedded  in  pQ(i)  is  the  mean  effect  of  the  attacks 
as  well  as  the  mean  effect  of  the  factor  levels.  We  also 
expect  the  variability  to  be  greater  in  the  attack  case  than 
in  the  no-attack  case. 

The  Experimental.  Das ism 

The  experimental  design  needs  to  isolate  each  main 
effect  and  two-way  interaction  so  that  the  significance  of 
each  can  be  determined.  A  Resolution  V  design  is  required  so 
that  no  main  effects  or  two-way  interactions  are  confounded 
with  one  another.  A  full  factorial  design  requires  210  or 
1024  separate  simulation  runs  for  each  case  (i.e.,  attack  and 
no-attack)  for  a  total  of  2048  runs.  However,  since  we  are 
assuming  three-way  and  higher  order  interactions  to  be 
negligible,  we  can  and  should  use  a  fractional  factorial 
design.  From  McLean  and  Anderson  (1984),  we  selected  a  1/8 
replication  of  a  10  factor  design  (nine  resources  plus 
attrition  of  our  aircraft)  at  two  levels  each  (Figure  3.2) . 
This  design  requires  128  treatment  combinations. 

We  also  want  to  estimate  the  impact  of  air  base  attacks 
on  sortie  generation  so  this  effect  must  be  isolated  if 
possible.  Our  approach  is  to  perform  the  experiment  without 
attack,  and  then  repeat  it  with  attacks  included.  The 
results  then  can  be  compared;  since  the  factor  treatments  in 
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each  sub-experiment  are  the  same,  we  should  be  able  to 
isolate  the  effects  of  the  air  base  attacks.  This  approach 
results  in  a  total  of  256  separate  simulation  runs,  128  for 
each  case,  of  thirty  days  each.  We  should  note  here  that 
this  approach  restricts  randomization  since  a  treatment 
(i.e.,  factor  levels)  is  input  to  TSAR  and  run  for  30 
consecutive  days.  This  makes  the  daily  runs  correlated  for 
each  treatment.  The  alternative,  running  each  day 
independently  within  each  treatment,  would  require  7680  runs 
which  is  prohibitive. 

Randomness  in  the  logistics  processes  in  TSAR  and  the 
sensitivities  of  results  due  to  randomness  of  TSARINA  attacks 
led  us  to  use  a  blocking  scheme  to  reduce  the  variance  as 
well  as  to  help  ensure  that  the  results  did  not  depend  on  a 
single  scenario.  Randomness  is  a  key  component  of  simulation 
models  and  the  results  are  often  related  to  the  streams  of 
random  numbers  used  in  the  model.  By  blocking  based  on 
random  numbers,  we  can  usually  reduce  the  experimental  error 
as  compared  to  a  completely  randomized  design. 

A  major  consideration  is  the  number  of  blocks  to  use  in 
the  experimental  design.  The  choice  involves  tradeoffs, 
especially  in  our  interest  in  reducing  variance  as  well  as  in 
capturing  a  representative  sample  space.  On  one  hand,  no 
blocking  allows  the  widest  possible  inference  space  with  a 
different  random  version  of  an  attack  for  each  treatment 
combination.  However,  this  does  not  help  to  reduce  variance 
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as  there  is  no  common  thread  between  the  runs  in  terms  of 
random  numbers.  On  the  other  extreme,  one  block  allows  the 
greatest  amount  of  commonality  because  all  treatments  use 
exactly  the  same  attack.  However,  this  restricts  the 
inference  space  since  we  have  only  a  specific  attack  based  on 
a  single,  specific  stream  of  random  numbers.  Previous 
research  uses  both  extremes  where  Folkeson  et  al.  (1986)  use 
a  single  attack  for  all  resource  combinations  and  Folkeson 
(1988)  replicates  a  design  point  with  thirty  random  versions 
of  an  attack.  Due  to  the  size  of  our  experiment,  this  latter 
approach  is  not  practical  in  terms  of  computer  run  times. 

Thus  we  sought  an  approach  that  would  yield  variance 
reduction  through  blocking  and  still  have  a  representative 
inference  space  that  accounts  for  randomness  in  the  attacks. 
As  a  result  of  these  tradeoffs  we  selected  a  design  with  8 
blocks . 

In  the  no-attack  case,  each  block  is  defined  by  a 
different  starting  seed  (randomly  selected)  for  the  TSAR 
random  number  stream.  The  random  numbers  for  all  random 
events  are  drawn  from  this  stream.  Most  of  the  random  events 
in  TSAR  concern  the  operation  of  the  aircraft,  i.e.,  which 
systems  and  components  on  the  aircraft  malfunction  and  need 
to  be  repaired.  The  flow  of  random  numbers  is  not  congruent 
from  run  to  run  because  malfunctions  cause  additional  random 
numbers  to  be  used.  For  example,  in  one  run  random  number  A 
may  determine  whether  the  engine  on  Aircraft  #1  requires 
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maintenance;  if  so,  random  number  B  will  be  used  to 
determine  what  component  on  the  engine  failed.  Suppose 
random  number  A  had  indicated  that  the  engine  on  Aircraft  #1 
required  no  maintenance.  Then  random  number  B  will  be  used 
to  either  check  another  system  or  another  aircraft.  With  72 
aircraft  and  81  systems/subsystems  which  can  malfunction,  it 
is  easy  to  see  that  the  the  random  number  usage  is  not  going 
to  be  congruent  from  one  run  to  another.  Other  random 
numbers  are  used  to  determine  which  aircraft  are  attrited  by 
the  enemy  and  the  time  to  repair  battle-damaged  aircraft. 
Although  we  have  this  obvious  incongruency  run  to  run,  each 
block's  results  do  have  a  common  starting  point  in  the  form 
of  the  starting  random  seed,  and  thus  we  can  determine  the 
blocking  effect  for  each  daily  model.  Because  of  the 
incongruent  use  of  random  numbers  in  TSAR,  we  do  not  expect  a 
large  blocking  effect. 

In  the  attack  case,  each  block  has  the  same  attack  in 
terms  of  targets  and  number  of  attacking  aircraft,  but  each 
has  a  different  randomly  selected  starting  seed  within  the 
TSARINA  model.  This  gives  us  eight  different  random 
variations  of  the  same  attack.  Random  numbers  are  used  to 
allow  for  randomness  in  the  results  of  the  intended  attack. 
For  example,  each  attacking  aircraft  has  a  Desired  Mean  Point 
of  Impact  (DMPI)  which  is  where  the  pilot  wants  the  bomb(s) 
to  land.  However  the  model  allows  for  dispersion  around  each 
DMPI  to  be  determined  by  random  numbers.  Hence  bombs  may 
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land  short,  long,  left,  or  right  of  the  DMPI.  Further,  the 
ballistic  dispersion  and  the  reliability  of  each  bomb  is 
determined  by  random  numbers  from  the  stream.  As  a  result, 
each  attack  is  going  to  be  different  based  on  the  starting 
seed  for  the  TSARINA  random  number  stream.  The  use  of  random 
numbers  in  TSARINA  would  be  congruent  case  to  case  except 
that  a  probability  of  arrival  is  also  randomly  checked  for 
each  attacker.  If  an  attacker  is  attrited,  the  model  does 
not  check  its  bombs  for  impact  point  and  reliability.  TSAR 
seeds  are  also  repeated  for  each  block  so  that  the  only  major 
difference  between  comparable  blocks  in  the  two  sub¬ 
experiments  is  the  attack.  Figure  3.3  depicts  the  blocking 
scheme  used  with  regard  to  random  number  streams.  With  the 
inclusion  of  blocking,  we  need  to  restate  the  metamodels  to 
be  estimated. 

The  Revised  Metamcdels  (With  Blocking^ 

With  blocking  in  our  experimental  design,  we  need  to  add 
another  factor  to  each  metamodel  that  captures  the  effect  of 
the  various  random  number  streams  in  the  TSAR  and  TSARINA 
models.  The  intent  is  to  explicitly  account  for  variability 
due  to  the  block  effects  described  above  and  thereby  reduce 
experimental  error  and  make  the  experiment  more  powerful. 

The_Me.tamodel  for  the  No  Attack  Case 

10  9  10 

Si  =  Po(i)  +  I  Pj(i)Xij  +  S  Z  pjk(i)XijXik 

j=l  j=l  k=j+l 
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+  Bu(i)  +  e(i)  (3.3) 

where  =  the  number  of  sorties  flown  on  Day  i 
=  level  of  factor  j  on  Day  1 

(i)  reflects  the  random  effect  on  Day  i  due  to 

the  random  number  streams  in  TSAR,  where 
Bn(i)  ~  N(0,Og) 

£(i)  reflects  the  experimental  error,  where 

2 

£(i)  ~  N(0,Oe) 


*  *  10  *  9  10  * 

Si  -  P0(i)  +  X  p4<i)Xij  +  X  X  p,.(i)XijXik 
j-1  J  j-1  k= j+1  J 

+  B*x (i)  +  e* (i)  (3.4) 


★ 

where  =  the  number  of  sorties  flown  on  Day  i 
X^j  =  level  of  factor  j  on  Day  1 
★ 

B^  (i)  reflects  the  random  effect  on  Day  i  due  to 

random  number  streams  in  TSAR  and  TSARINA,  where 
3n(i)  ~  N  ( 0 ,  Og  * ) 

* 

£  (i)  reflects  the  experimental  error,  where 

*  2 
£  (i)  -  N(0 ,Og*) 
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Assessing  the  Research  Objectives 
Four  research  objectives  are  identified  above  which  are 
the  focus  of  this  work.  Below  we  discuss  how  we  measure  the 
accomplishment  of  each  objective. 

The  Effectiveness,  of  the  Experimental  D.esigjg 

A  major  objective  of  this  research  is  to  apply  an 
experimental  design  that  reduces  the  number  of  runs  as  much 
as  possible  while  achieving  acceptable  levels  of  experimental 
error.  To  assess  the  extent  to  which  this  objective  has  been 
achieved,  we  estimate  the  effectiveness  of  the  blocking 
scheme  in  terms  of  design  efficiency  and  variance  reduction. 

Relative  efficiency  of  blocking,  compared  to  complete 
randomization  is  defined  by  Neter  et  al.  (1985)  as: 

2 

or (completely  randomized  design) 

E  =  - 2 - 

^(randomized  block  design) 

The  MSe  =  SSg/dfe  for  the  randomized  block  design  is  an 

2 

unbiased  estimator  of  Gb  (Neter  et  al.,  1985).  Further,  an 

2 

unbiased  estimator  of  Gr  is  obtained  by  pooling  the  blocks 
and  error  components: 

7 

I  SS(3ij)  +  SSe 

*2  i^l _ 

ar  ”  dfB  +  dfe 


Thus  we  have  relative  efficiency 
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j 


E 


7 

t  I  SS(Bij)  +  SSe]  /  [7  +  df£] 

i=1 _ 

SSe  /  dfe 


(3.5) 


This  measure  of  relative  efficiency  indicates  how  much 
replication  must  be  done  with  a  completely  randomized  design 
to  achieve  the  same  precision  in  the  variance  of  significant 
factors  as  obtained  with  the  blocking  design. 

Percent  variance  reduction  follows  from  the  definition 
of  efficiency.  Thus, 


%  variance  reduction  = 


100 


[old  variance  -  new  variance] 
[old  variance) 


=  100 


=  100 


(  I  SS(Bij)+SSe) 

•HI  SSe 

(dfB+dfE)  "  dfE 


(  I  SS(Bij)+SSe) 
HI 


(dfs+dfe) 


(3.6) 


Estimating  the  Daily  Metamodels 

Another  major  goal  of  this  research  is  to  estimate 
simpler  metamodels  from  the  detailed  TSAR  and  TSARINA 
simulation  models.  Based  on  the  above  experimental  design, 
we  have  a  total  of  3840  observations  for  each  case  (attack 
and  no-attack) .  In  other  words,  we  have  128  responses  for 
each  day,  one  for  each  treatment  or  resource  combination 
found  in  the  design.  Before  deriving  metamodels  from  these 
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data,  we  first  test  whether  the  populations  of  the  response 
variable  are  normally  distributed.  Next,  based  on  the  output 
data,  we  estimate  the  daily  metamodels,  and  then  evaluate 
their  fit.  Lastly,  we  compute  confidence  intervals  for  the 
population  mean  at  certain  design  points. 

Using  the  Shaoiro-Wilk  test  for  normality,  we  examine 
each  case,  attack  and  no-attack,  to  determine  whether  the 
response  variable,  sorties  flown,  follows  a  normal 
distribution.  The  number  of  sorties  flown  on  each  day  for 
each  design  point  has  its  own  population  characteristics;  we 
therefore  cannot  combine  the  128  daily  sample  data  to  test 
for  normality.  However,  we  can  test  for  the  normality  of  the 
population  of  responses  at  a  specific  design  point. 

Normality  can  be  tested  at  several  design  points  and  then  we 
can  possibly  infer  the  same  results  for  the  remaining  design 
points.  Here  we  will  select  two  design  points,  (a)  all 
factors  high  and  (b)  all  factors  low,  and  run  20  independent 
replications  for  each  case  (i.e.,  with  and  without  attacks) 
for  sorties  flown  on  Days  i  =  1,5,30.  These  two  design 
points  ought  to  represent  the  upper  and  lower  bounds, 
respectively,  of  our  inference  space. 

For  design  point  (a),  let  denote  the  number  of 

sorties  flown  on  the  i  day  of  the  j  independent 
replication,  1  <  i  <  30,  1  <  j  <  20.  Similarly  define 
for  the  point  (b) .  Then  we  apply  the  Shapiro-Wilk  normality 
test  (Anderson  and  McLean,  1974)  first  to  the  data  set 
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(S#  :  j  -  1,  20} 

then  to  the  data  set 

{S5j  :  j=l,  .  .  . ,  20} 
and  then  to  the  data  set 

{S30 j  :  j  =  1,  ...,  20}. 

Similarly  we  test  s|^  for  normality  on  Days  i  =  1,  5,  30. 
This  gives  6  Shapiro-Wilk  test  statistics 
(k ) 

W^  for  k  =  a,  b;  i  =  1,  5,  30. 

If  a  is  the  overall  level  of  significance  for  the  normality 
test,  then  we  accept  the  null  hypothesis  of  normally 
distributed  daily  sortie  counts  if 

(k)  * 

Wi  >  Wa/6  for  1  =  1'  5'  30;  k  =  a,  b 

★ 

where  W^g  is  the  quantile  of  order  a/6  for  the  Shapiro-Wilk 
distribution  for  sample  size  20.  Thus  if  a  =  0.06, 
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0.10  level.  The  blocking  variables,  B1  through  B7,  are 

always  included  in  the  model  regardless  of  their  level  of 

significance.  The  resulting  metamodel  for  each  day 

identifies  the  factors  which  best  predict  the  level  of 

sorties  flown  for  that  day  given  the  level  of  significance 

for  retaining  a  variable  in  the  model. 

To  evaluate  the  metamodels,  we  examine  each  daily  model 

2 

with  regard  to  the  coefficient  of  multiple  determination  (R  ) 
to  see  how  well  the  independent  variables  in  the  reduced 
model  explain  the  variation  in  sorties. 

Confidence  intervals  are  next  estimated  for  the 

population  mean  of  specific  design  points. 

A  100(1-00%  confidence  interval  on  Jl)c(i),  the  mean 

number  of  sorties  flown  on  Day  i  for  the  kth  design  point  of 
the  no-attack  case,  is 

Xkfifl)  ±  ti-a/2<DF£(i))  .  MSe(i)lSk(X,X)-1^  (3.7) 

Similarly,  a  100(1-00%  confidence  interval  on  the  comparable 
* 

quantity  ( ± )  for  the  attack  case  is 


Zk£*(i)  ±  ti-a/2  (DFg*  (i) )  .  MSe*  (i)i)c  (X'X)  (3.8) 

Homogeneity  of  Variance.  Since  MS£*(i>  and  MS£(i)  are 

correlated  due  to  common  random  numbers,  we  cannot  claim  that 
MS£Mi)/MSe(i)  has  an  F-dist ribution  in  order  to  compare  the 
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variances  of  the  two  cases .  However,  we  can  compute  sample 
variances  based  on  the  20  independent  replications  we  made  at 
various  design  points  for  each  case.  Thus  for  the  no-attack 
case  we  have  sorties  flown 


S 


(k) 

ij  : 


1  <  j  <  20  for  Day  i  =  1,5,30 


and  design  point  k  =  a,b 


and  we  can  estimate  the  population  variance 


ft2 

&ik 


20 


(k) 


19  tSii 


—  /  vs  2 

;]  for  Day  i  =  1,5,30 


and  design  point  k  =  a,b 
Similarly  for  the  attack  case  we  have 

*  fkl 

Sij  :  1  <  j  <  20  for  Day  i  =  1,5,30 
and  design  point  k  =  a,b, 

and  we  can  estimate  the  oooulation  variance 


*2 


20 


(k)  T*(k),2 


°ik  =  19  S^tSij  "  si  J  for  Day  1  -  1,5,30 

and  design  point  k  =  a,b. 

From  these  data,  we  can  test  for  homogeneity  of  variance  a) 
within  each  case  and  b)  between  cases.  When  testing  within 


cases,  we  compare  results  for  design  point  (a)  to  results  for 
design  point  (b)  for  each  Day  i  =  1,5,30.  When  testing 
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between  cases,  we  compare  results  for  design  point  (k)  in  the 
no-attack  case  to  the  results  for  the  same  design  point  (k) 
in  the  attack  case  for  each  Day  i  =  1,5,30.  These  tests  are 
further  defined  below. 

For  testing  homogeneity  of  variance  within  each  case,  we 
have  the  following  for  each  Day  i  =  1,5,30  of  the  no-attack 
case : 


Hypotheses : 

H0  : 

°ia 

°ib 

2 

2 

Hi: 

°ia 

* 

°ib 

Test  Statistic: 

a2 

/ 

.2 

F  = 

aia 

aib 

Decision : 


Reject  Ho  if  F 
n2~l),  i.e., 
with  na  =  nb  = 
reject  Hq  if  F 


>  F  (a/2,  m-1, 

20  and  a  =  .05 

>  2.545. 


The  same  test  is  performed  for  the  attack  case  where  0^a 

2*2  2 
replaces  0ia  and  0^b  replaces  0^ . 

For  testing  homogeneity  of  variance  between  the  cases. 


we  have  the  following  for  each 
=  1, 5,30: 

Hypotheses : 

Test  Statistic: 


design  point  k  =  a,b  and  Day  i 


*2 

2 

K0:  0ik 

aik 

*2 

2 

H1 *  °ik 

T- 

°ik 

**2 

/ 

^2 

F  =  0. , 
ik 

aik 
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Decision : 


Reject  Hq  if  F  ^  2.545. 


A  third  research  objective  is  to  determine  if  attacks  on 
the  air  base  cause  a  significant  difference  in  flying 
performance.  To  evaluate  this,  we  would  like  to  use  a 
pairwise  comparison  of  the  attack  and  no-attack  cases  by 
treatment  and  by  day.  Thus  we  would  have  128  differences 

(one  for  each  design  point)  for  each  day, 

D.  =  S,  .  -  S~  .  j  =  1,2,  .  .  . ,  n 

1  ID  2j  j 

where  is  the  number  of  sorties  flown  with  jth  treatment 

in  the  no-attack  case  and  S0 .  is  the  number  of  sorties  flown 
with  jth  treatment  in  the  with  attack  case.  However,  the 
{Dj  :  1  <i  j  :£  n}  are  not  independent  since  the  design  uses 

common  random  numbers  across  all  n  design  points;  moreover 
they  are  not  identically  distributed  because  they  come  from 
different  design  points.  Thus  a  different  approach  is 
necessary . 

At  the  kth  design  point,  let  Xij (k)  denote  the 
corresponding  level  of  the  jth  factor  on  day  1;  and  let 


xk  ■  [1  Xi.,i(k)  ...  Xi,io(k)  Xi,i(k)  .Xi,2(k) 


...  Xir  9 (k)  .  Xir 10 (k) ] 


1  +  10  + 


9(10) 


56  terms 


(3.9) 


denote  the  overall  vector  of  factor  levels  so  that  the  no¬ 
attack  metamodel  can  be  written 


E [Si]  =  xkg(i)  (3.10) 

where  g (i)  ■  E3o(i)  3i (i)  • • -Pio (i)  3i,2(i)  . . -39. 10 (i) ] ' 

is  the  56-dimensional  vector  of  metamodel  coefficients.  Let 
n=128,  the  number  of  design  points;  let  b=S,  the  number  of 
blocks;  and  m=16,  the  size  (number  of  design  points)  of  each 
block.  Finally  let  W  =  |  |wrc|  )  denote  the  so-called  block 
incidence  matrix  defxned  as  follows: 


r  +  l 

if  design  point 

r 

falls  in 

wrc  — 

block  c  and  1  < 

c 

<  b-1 

-1 

if  design  point 

r 

falls  ?n 

(3.11) 

block  b  and  1  S 

r 

£  b-1 

L  0 

otherwise 

Thus  the  full  design  matrix  for  both  the  attack  and  no-atuack 
experiments  is 
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D 


w 


[s  S] 


(3.12) 


Let  Si  denote  the  nxl  vector  of  sorties  flown  on  day  i  for 
all  n  design  points,  and  let  £(i)  denote  the  corresponding 

nxl  vector  of  errors.  Let  3c(i)  denote  the  random  block 
effect  due  to  block  c  on  day  i,  1  <  c  <  b;  and  let 
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B  (i)  a 


Bi  (l) 

B2  (i) 

Bb-l (i) 

. -  -Bb-i (i)  “Bb (i) 


(3.13) 


denote  the  bxl  vector  of  block  effects.  The  overall 
metamodel  for  the  no-attack  case  can  be  compactly  summarized 
in  matrix  notation  as 


Si  =  X  g(i)  +  W  B(i)  +  £(i)  . 


(3.14) 


Now  since  Plan  8.10.16  on  page  265  of  Anderson  and 
McLean  (1984)  yields  a  "completely  orthogonal"  analysis 
(assuming  3-factor  and  higher-order  interactions  are 
negligible),  we  have: 


D'D  -  _  _  [X  S] 


X '  X  XJ  w 

W'X  W'W 


0  2mlb-i 


(3.15) 


This  implies  that  the  ordinary  least  squares  estimation  of 
B ( i)  is 


B  (i)  =  (xjx)"1  x'Si 
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=  (X'X)"1  X'  [  Xg  ( i  >  +  WB(i)  +  g(i) 

=  g(i)  +  (X'X)"1  X'g(i)  (3.16) 

since  X_'W  =  0  . 

From  (3.16)  it  follows  that 

Cov[$(i)]  =  E{  [X'X)  _1X’  g  (i)  ]  [  (X'X)  _1XJ  g  (i)  ]  '  } 

=  (X'X) _1X'E{g (i)E'  (i)  }X(X'X) -1 
,  2 

=  (X'X)-iX' {Oe(i  In)  X(X’X)-1 
2 

=  ag(i)  (XJX)-I  (3.17) 

Combining  (3.10)  and  (3.17),  we  see  that  the  minimum  variance 

unbiased  estimator  of  |lk(i),  the  mean  number  of  sorties  flown 

on  Day  i  at  the  kth  design  point,  is 

£k(i>  =  xjcg(i)  (3.18) 

=>  Var  [jlk  (i)  ]  =  xkCov[§(i)  ]  x '  k 

2 

=  o£(i)  xk(XJX)-i.xJk  .  (3.19) 

Similarly,  for  the  attack  case  we  have 

S*  =  XpjMi)  +  W&Mi)  +  e*(i)f  (3.20) 

ff*(i)  =  (XJX)  _1xj  s’ 


=  Pj»(i)  +  (x’x) -1x,e*  (i) 


(3.21) 
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/v  2 

Cov[£*(i)]  =  <Je(i)  (X'X)"1  ;  (3.22) 

*• 

and  the  minimum  variance  unbiased  estimator  of  |lk(i),  the 
mean  number  of  sorties  flown  on  Day  i  at  the  kth  design  point, 
is 

£k  (i)  =  xk[f*  (i)  (3.23) 

with 

^  2 

Var[il*(i)]  =  aeMi)x)c(XJX)-1xJk  .  (3.24) 


a  * 

We  want  to  use  ^k(i) 


£k(i)  to  test  the  null  hypothesis 


H0  :  }ik  (i)  =  )Ik  (i)  (3.25) 

that  there  is  no  difference  in  the  mean  number  of  sorties 
flown  on  Day  i  with  and  without  attacks  versus  the 
alternative  hypothesis 

Hi:  }lk  ( i)  <  (lk  (i)  (3.26) 


that  fewer  sorties  are  flown  in  the  attack  case. 

Now  if  we  had  run  the  attack  and  no-attack  cases 
independently,  then  we  would  have  had 


Var[jlk(i)  -  £k(in  =Var[£k(i)]  +Var[£k(i)] 


(3.27) 


Because  we  used  the  same  random  numbers  for  each  case,  we 
have  instead 


Var  [Sk(i) 


Mk  ( i )  ]  =  Var[jxk(i)]  +Var[^k(i)] 
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A  * 


-  2Cov[nk(i) ,  ( i ) ]  ; 


(3.28) 


and  if  the  use  of  common  random  numbers  has  been  effective  in 
sharpening  the  comparison  of  the  attack  and  no-attack  cases, 
we  will  have 


A  * 


Cov[|Ak(i)  ,  Jlk(i)  ]  >0 


(3.29) 


so  that  the  right-hand-side  of  (3.28)  is  less  than  the  right- 
hand-side  of  (3.27). 


case,  we  have 


/ no-attac 

From  the  results  of  analyzing  the  data  for  the  y  attack 
/  SSg  /  i )  \ 

(sse*{*)j  the  error  sum  of  squares  for  the 
/  (3.14)  \  fDFE(i)  N 

metamodel  ^  (3  20)  /  on  Da^  anc*  I  dfe*  { i )  J  t^ie 

corresponding  degrees  of  freedom  for  error  so  that  the  mean 
square  for  error  is 

f  MSc(i)  =  SS8(i)/DF8(i)  ^ 
t  MSe*(i)  =  SSe*(i)/DFe*(i)  J 

To  test  Hq  (3.25)  we  compute 


ix*(i)-fk(i) 


to  = 


*\j  [MSe(i,+MSt*(i)]  [xk(X'X)"1xk] 


XktPj*  (i)-B(i)  1 


V 


[MSe(i)+MSeMi)  ]  [xk(X'X)-lxkJ 


(3.30) 


and  we  reject  Ho  in  favor  of  Hi  at  the  a  level  of 
sianificar.ce  if 
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to  k  ta  (Veff )  t  (3.31) 

where  ta(veff)  is  the  quantile  of  order  a  for  a  Student  t- 
distribution  with  the  so-called  "effective  degrees  of 
freedom"  (Welch,  1947) : 


Veff  = 


[MS£  (j)  +MSe*  (j)  )2 


•  [MSe(i)]2 
.[DFe<i)+2] 


[MSeMi)]2 
[DFe*  (i)  +2  ] 


-  2 


(3.32) 


Because  of  (3.31)  and  (3.32),  this  test  is  conservative., 

A  *  A 

that  is,  a  difference  ( ± )  -  (l^(i)  that  is  reported  to  be 

significant  at  the  a,  level  of  significance  is  actually 

significant  at  an  even  lower  level;  thus,  significant 

differences  are  understated.  Finally,  a  conservative  100(1- 

* 

a)  %  confidence  interval  on  the  difference  fi^i)  “  ^(i)  is 
Xk  [{£*  (i)  -j3  (i)  ]  ± 

ti-a/2(veff)  '  [MSt(i)  +MSE*  (i)  ]  [x*  (X*  X)  ~1Xjc]  (3.33) 


Identifying  Key  Resources  Over  Time 

The  last  major  research  objective  is  to  identify  key 
resources  and  interactions  over  a  thirty-day  time  period  with 
and  without:  attacks.  We  have  estimated  thirty  separate  daily 
models  and  now  we  want  to  see  if  there  are  resource  trends 
across  these  models.  This  evaluation  is  largely  subjective 
based  on  the  beta  coefficients  of  significant  variables  in 
the  estimated  metamodels.  We  examine  any  trends  in  the 
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coefficients  from  day  to  day.  We  also  use  the  metamodels  as 
predictive  tools  to  estimate  the  difference  in  the  expected 
number  of  sorties  flown  when  a  single  variable  moves  from  the 
low  level  to  the  high  level. 
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CHAPTER  IV  -  OVERALL  EVALUATION  OF  THE  EXPERIMENTAL  DESIGN 

Results  are  divided  between  Chapters  IV  and  V.  This 
chapter  focuses  on  the  experimental  design  and  the 
statistical  analysis  of  the  metamodels.  Chapter  V  continues 
the  evaluation  of  the  results  by  interpreting  the  metamodels 
in  terms  of  resource  or  factor  importance  to  the  sortie 
generation  process.  Here  we  first  examine  how  well  our 
design  performed  in  terms  of  variance  reduction  and  relative 
efficiency.  We  then  look  at  the  daily  predictive  metamodels 
derived  from  the  simulation  data  and  test  for  normality. 
Confidence  intervals  for  the  mean  response  are  also  computed 
and  we  examine  the  sample  variances  for  several  cases.  Next 
we  evaluate  the  predictive  power  of  the  regression 
expressions.  Finally  we  test  for  significant  differences 
between  the  attack  and  no-attack  cases . 

The  Effectiveness  of  the  Experimental  Desicn 
Cur  first  research  object!'.0  in  this  thesis  is  the 
development  of  a  statistically  controlled  experimental  desig 
which  allows  us  to  estimate  metamodels  from  a  relatively 
small  number  of  runs  of  a  large-scale  simulation  model.  Thi 
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must  be  done  with  acceptable  levels  of  estimated  error 
through  the  use  of  variance  reduction  techniques. 

Variance  Reduction  and  Design  Efficiency 

The  experimental  design  has  mixed  results  in  terms  of 
reducing  variance  by  blocking  based  on  common  random  numbers . 
Tables  4.1  and  4.2  show  the  results  for  the  no-attack  and 
attack  cases  respectively. 

The.  _No- At  tack  Case.  The  success  of  the  variance 
reduction  technique  in  this  case  ranges  from  a  high  of  43.72% 
reduction  for  the  Day  1  results  to  -4.91%  for  reduction  Day 
22.  The  average  variance  reduction  across  all  thirty  models 
is  only  5.68%.  Thus  it  appears  that  the  blocking  scheme 
based  solely  on  random  numbers  in  the  TSAR  mode]  is  not  very 
effective.  In  light  of  the  complexity  of  the  model,  these 
results  are  understandable.  Each  30-day  run  within  a  block 
begins  Day  1  with  the  same  seed  for  the  random  number  stream. 
From  this  common  beginning,  the  runs  begin  to  diverge  in 
tneir  use  of  random  numbers,  some  runs  using  more  random 
numbers  than  others  depending  on  such  events  as  aircraft 
losses  due  to  attrition,  aircraft  battle  damage,  and  failures 
within  any  of  the  81  aircraft  systems/subsystems .  As  a 
result  of  this  random  divergence  in  the  use  of  random  numbers 
- rom  tne  common  stream,  runs  after  the  very  beginning  of  Day 
1  have  little  or  no  congruence.  Thus,  it  is  not  surprising 
that  we  find  our  best  results  on  Day  1.  Table  4.3  shows  the 


No-Attack  Case  —  Variance  Reduction  and  Relative  Efficiency  Due  to  Blocking 


69 


no— o©NQ©<\i©*,N©©-*©oin©N^©in(v©©©©©rgg 
I^QfM^OO^Off'O^O'OOff'O-OH^Off'OHOOO'O'OO 


nNa~®~CDCD-«'eCD©-'vr>(r*cgCDm*-T®cj--*'r"f-*-«a',eir>ro 
H  NCDT\fl’tnNmOvflTfMOCDNH(Mnp1jN^ff\oa30'Nvi3-'Vl)O 

ii  m  o  ©  ~  ©  a  m  ci  ~  ©  ©  -J  ©  d  o  o'  m  ©  V  ©  ©  V  d  m  ci  in  ~  rJ  -*  q 
II  T  -•  -«  —  I  II  T  I  H  H  I  ~  II  T 


*-s  ii  m^fDHinoinmNN-HoyoininNNcaNCDt^ot^iflmo 
ui/6ao-oooooo--‘^^^Dodoo©ooo^oooo^ 


^^^Vi)-^rMCT)inofnoo^,^©'t-N©'<©(Ni©©'r©ino©fM© 

d  d  ©  <tj  d  V  ©  ©  rJ  in  in  rg  cv  ®  ©  V  a  cu  rv  in  <r  ©  ©  ©  d  ©  <e  ©  o 
rgr\i©Ntn’*©-«T’T©©©<5,*‘PNQ!nNint©^©©Q  —  ©q 
©©©,e©-«N©©^,c©CDin**«Nino©©'cin,cNin*-«©N©ni 
f\»©m©©©0N0©-«rg©©©©in©rg©inNin©0'r©-'cN^ 


©N©©©©©O^©©Of\J©NN'*^O©N~Oi-'ONO0J©© 

d  m  r\]  d  n  rv!  in  V  ©  ©  ©  in  n  d  J  d  o*  ©  •-  ©*  d  i/i  o  s'  d  in  ©  d 
in  ©  ^  in  o  in  tn,r^iQ©p  -«  +  -«  n  -«  ^  *e  ©  rg  ©  © 

(\i  m  — '  •-«  m  n  Hon  t  rg  — •  ©  ©  ^  *■* 


ooa'O'tfH^fONONcnHajHfnoiom^o^moNto'iflifl 

R  £  2  8  ™  2  i  8  $  2  ^ y5 n n d q d d d p d n 


©©NNOQ^-*-*  a  ©  N 

©  ft -* 'r  rn  ©  ©  in  n  rg  rg 

CNJ  ©  -•  ©  ▼ 


tfNOvflvflinD^-o^a'*flNO'-o'g,in(M’rg}fn\fliMOo^o 

©drg©f\idNdd*d©inddN©©NdNddddddVdr\i 
m  —  ~  in  in  n  r  (\  m  va  g)  ~«©©g)>rN©©©inmf\iinQg 

rg  ©  in  C4  rg  ©  —  n  rg  ©  ©  ▼  n  o© 


©©fgin©  —  ©T-«©©<N©Nrg©inNrin~©  —  ©rorg^-NO 

--ocgNinQ©©©©'™  ©  indN©NNNN©dN©tnd  —  in©© 
©  in  ©  in  eg  n  -«  t  in  *-  ©  n  —  ©No©o©©inrjN©©©N 
©  — *  -+  ©  in  g  n  ©  ~«  rg  ©  •*  o  in  ©  **  t 


rg  n  ©  •*  ©  —  arg©©©rgN©©©N©©©Nin©Q©<*,*'0-»N 

ir-oogfno^inri^Off'NHrjonMniMnfflinNiftgitON 
©  ©  in  in  ©  in  ©  **  ©  ©  rg  ©  **  ©  •**  ©  eg  ©  ▼  ©  n 

▼  ©  in  ^  rg  —  ©  -*  *-•  **  *-•  ©o  in 


—  ©©<T©inTryj^©m©o©©©©©©©©©©©©^rgo  —  rs. 

N©inin©©©Nd©rJtn©rjdNtn©rvd  —  rg  —  d©rg©ddd 
Nooo  n  —  ©  ©  ©  in  in  ©  in  ©  o  ©  rg  in  ©  —  ©  ©  ft  ©  — 


®  rg  ©  t  —  —  ©oqnoo  —  orgiNrj*©o©©©  —  no©o©w 
gj  ▼  n  o  Mn  o  d  ^  cn  m  1 1  m  cm  n  o  o's  n  h  o  o  (tJin  (ndoN 

C\l  ©  T  —  CJ  ©  — «  r\l  N^tOCOMtlt  ^ONNttfflt- 

—  CM  ©  O  ©  ©  N  ©  «r  IM  ©NO© 

—  —  —  eg  -* 


■  NNNNNNNM 


70 


ui>-  if  —  mrM©N  —  ©©©©<M0©©©©CMrg©fg(M©o©©  —  ©  ©  “*  © 
:>  y  ii  ©©N©©©©rMM’©M,©©©©©©M,M,©N©in©©©©©©© 

hu  ii  —  ©cm---^  —  —  -  —  -  —  ^  —  -  -  —  —  —  -  —  —  — 

<T  —  ll 
JU  II 
Q  *-•  H 
5  u.  « 

ui  z  ii  ©o©oN©ocgK-*aa©©cMcgoc\i©ffl©©<v<MfM0,'rm  —  o 
o  a  n  ©©  —  ©x©N©©a©©©©©(Mcg'*©<M©©M>N©©fN.©  —  — 

z  *-*  ii  •  . . ........... . 

c  b-  it  n  —  ©©©mmm  —  mmvo©©©N-©©fg(p  —  ®©o©o-©©©o- 
~  u  ii  tcoiijMvwfnHtDNfjmMftmfnftNmfntmmtnitntfrjnHn 


u 

U  H  —  D  —  -  -  —  —  —  —  -  oO  —  5  —  -  OOOOOOOOO  —  OOOO 


cDaJmN-mmvinofflotma'ioc^fVHMt-tDDn^’rsD 

m  rg  ©’  ©  V  ©  rg  ©  d  o!  is.*  o  ©  a'ooVdoHt^TNmiMuiN^O' 
©rg©'erv©cDrgT©aN'ca*-N©N©CD<<cNfgo©©N©a>©© 
©'£©Q©©©©fMN©©,,r',“*©,,*©o*©rw©  —  QN*,©©©Ow  — 

—  ©©©inrnrvnuirgrg-c  —  o<\i©©o©o©  —  ocm  —  rg  —  *<01  — 

—  eg©'*©  —  rva'N<D©m©©©©©©©Tm©^rm©©rgrg{Mrg 


©o  —  ©  —  ©owcg  —  ©•*©®fg©©o©-*rffl©N0(M©'*©© 

VmNmaia'vDTN-fnvo^NMnvflOff'vflmN-^inw^wNN 
fgrgQ©-rr)g©©N©rgQ©NNN©©©©©©  —  ©CgOCM^-* 

rg  —  ©  cm  eg  ©  —  n  ©  ©  o  t  in  —  n  m  cm  in  x-  in  —  ©  o  —  ©  ©  ©  o.  in 
cm  cm  —  (ni  —  —  —  —  cm  cm  r\»  rg  —  cm  —  —  —  —  —  —  — 

OOOHfflCMtMDtdlOOtHtmOOtOMOlOinO'NHttN 


n  ©  © 

O'  <m  m  ■»■  m  ■. 

©  -  o  ©  rg  © 

©  -  0 

^  co  in  ©  5 

DOCM©N©©  —  Q 

o  m  o 

©  o  ©  -  ©  (Nl 

•»  0)  f 

«*  ©  ©  N  c 

D  O  f  ©  -  <NJ  N  —  © 

no© 

O  ~  ro  \o  •->  ■* 

-  ©  ©  ©  ©  r 

g  ©  ©  ©  — 

-  ©  T 

©  -  -cm 

—  —  —  CM 

— 

ii(T)omaJvflffl(nNoaja>ooifiONONHvfltintcDNO'*-Nto 

in  »i  0©©CM©Vin©ro©rg*V-V’'rVNN®<e©©©©®©®©d 

m  i  rvfs-rjininrgrgTfvTajfs.  ^oMnNNff'fflCDin-JCjTOTvflCD^N 
^  ii  —  ©  ©  m  □i/:oNNmNONNMinomoqN('JCDOMOoo'-«(N 

vo  ii  in©  in  t©©t©©t©^©©©t©tcm  —  cm  —  cmcm  —  —  — 

io  ii  © 

ii 

ii©fg0©©0©CMfM-©*  —  ^cg©T^©-©a©-fM(M©oo© 

^  ii . . . . . 

▼  n  ©  ©  o.  —  Tff>©©©o©©©  —  r*-©©©oo©©©  —  N^grwrg 
a  ii  t^tfiofgTfNjoc'rtN-NaNM^TinoHSN^NNfflom 
v/||  —  ©©©t©©©t©  Nffip-WR-OtpffWntNONNN 
vo  ii  t  —  (Nl  —  rg  —  rg  rg  —  —  —  —  Cg  —  fg  —  —  —  — 

vn  ii  — 

ii 

ii©cmo  —  ©©nno©©(n©©©o<\i(m®  —  ©  —  n.©^’’*©**©© 

^  ii  • . ....... . . 

©  u  mf;tf^ifloo'oofvcDrjo^^NNOintinm-(\jo-'-'i)Tg 
©  ii  o-5p^y!)NtwtnMvoptNm-ntN^ff'wmNc^^ffl 
^  II  NiNO®\l)\flCDiitNCDt^NOCDflNN|T)fioWn)COO&'N3 
in  ii  fninoo-T^nncminTinrjrg^  —  cmcmcmcmwcmcm©  —  —  — 
in  ii  -  —  —  in 

n 

noo©©fM^©  —  rg©©<r©©©o  —  rg©©o©©©o»rg©©fg© 

'N  II . . . . 

an  oot-otn  oin  oip  t  g\  t  ir)  Tino'iny  t  inmtNNNoit 

ii  —  ©  *■  t  r\i  —  o  cm  ©  n  o  rg  n  —  m  ■*  o  —  —  ©  —  in 

^iirg©in—  o.  ©  —  —  —  —  n  © 

©  ii  ©  n  ©  — 

©  ii  © 

ii 

noomofnsff"r(\iiriijr--^i%it'ff'in-ogj-ininaog(MfN 

ii  . . . 

—  ii  <voo©o,r©©o»T,®©©N.orM©©©'*,N- ▼©©©©©©©* 
©  ii  ©ocoinmo\©N  —  rgrg©go©o^,^o©CMOCT'(r*©rs.r'.©© 
w  11  aNNNff'tOC'tmONtOlMC'nNNOOff'^mtQNlfllfl^ 
©  II  —  —  Nff!OBNTU)\D<ONOinNyOONinO\flll)N\DiAlflt 


-Nntin^NOff'O-MnTiftONOff'O-rgnTinoNO^o 
—  —  —  —  —  —  —  —  —  —  (MCgCMCMCMCMCMCMiN.-g© 


No-Attack  Case  —  Computer  Run  Times  (Seconds) 


71 


CB 

u. 

u 

a 

CD 


V 

u 

o 

_J 

CO 


U 

O 

_J 

CD 

in 

v: 

a 

o 

_i 

CD 


V 

U 

o 

d 


m 

* 

o 

a 

_i 

CD 

CM 

■u 

u 

o 

-I 

m 


v 

o 

o 

_i 

CD 


Z 

Ul 

r 

k- 

<E 

UJ 

CSC 


CD(M03(MCDODCDaiB'CO(MQ3aa3'<- 

<B'^O'^V(T)CDU)'rr0ff'iBNNff)N 

a  o'  V  o'  m'  rn  oJ  ®  o'  in  n  o  ro‘  im  o’  in 
CDroororootn  —  inaro'g-CMroyin 
CMlMCM-*CMCM^CMCMCM-<(MCMCMCM.-t 


■'rOOOrofflroCMrorororoCMroCDCO 

NO'CD'fUJNmNNS'niMUINliJN 

o'  ro  o  d  ro  o'  n  o'  n  in  cm  ~  in  ro  ro  m 
cMco-q-inroro^inNiM-imaro-'i-a 
cM^cMCM^cMroiM-iiMroiMCM^cMro 


roffiCMCDM-Ororororo'4-rooCMaCM 

U]U|H<fHHQ(\|(r|On(M«HS'f 


nofflfflfOff'N'finD'rinMnN 

ND\NinoMTi\mifi(\njOinaiffl'f 

CMIM'-<CMCMCMCM~<CMIMCM~<<M»-‘^IM 


(MO®®-C®'<fmOffl®T(Mm®^ 

NMmHinni'tff'nQinootNQ 


O' 

® 

® 

ro 

O 

CM 

_1 

ro 

_ 

ro 

•“4 

O' 

in 

a> 

kO 

O' 

rs. 

a 

N 

O' 

ro 

ro 

ro 

r—4 

*“4 

CM 

CM 

•H 

r*H 

(M 

CM 

*— < 

CM 

•— < 

CM 

CM 

o 

o 

VO 

® 

CM 

U> 

-If 

a 

O 

CM 

o 

ro 

CM 

ro 

T 

O' 

® 

in 

*— 4 

in 

f— ♦ 

\0 

CM 

ro 

ro 

ro 

CD 

• 

• 

■ 

• 

i 

• 

• 

• 

« 

• 

1 

• 

■ 

* 

< 

< 

in 

o 

ro 

ro 

vO 

in 

M- 

m 

CM 

*■ 

ro 

ro 

® 

m 

o 

CM 

o 

CM 

O' 

■*■ 

N 

O 

\D 

in 

«-4 

ro 

CM 

m 

CM 

CM 

CM 

CM 

CM 

H 

CM 

CM 

H 

CM 

CM 

CM 

O 

m- 

ro 

CM 

a 

o 

CM 

■<■ 

Osi 

r 

® 

ro 

■4- 

o 

ro 

O' 

in 

in 

® 

N 

ro 

® 

CM 

ro 

O 

m 

ro 

ro 

ro 

ro 

M 

t 

-kJ 

l 

•H 

€ 

N 

m 


n  ro 
is. 

■v 


M 

C 

0 

0 

Of 

M 


CM  N 
O  N 


cMror‘»CMror^mo-4-ro~roror'-ro,'f 

M’fiCO'ff'tDO'NONmifiOtCOI!' 


inro-^ror'-ar^TCMC'.cnorocMiMro 

in  ro  in  in  ro  ®*  ro  V  cm  cm  —  d  ~  o  in 
M-rocMi^rocMro  —  in^ocn-raiNT 
CM-*cMCM-«iMiM<M~<MrorM<M-<iM(M 


M-CMorocMro-4-roro'rrororororocM 

o'N-N'omNu3Qoi\a-ao'(v 

-  —  cM-g-roV-g-d-Tro^o'CMin  Vcm 

infflNOOONOOQNC'HXH^ 

CM  CM  CM  ~  <D  <M  — >  <NJ  CsJ  CM  — >  -<  CM  ®  CM  - 


-«iMro<4-inroNroroo-<tMro'vinro 


cm  ro 

T  CM 
®  (M 
® 
cm 


u 
•  •  z 

(!)  t-4 

Z  K 
I-  Ul 

ro 
_j  c 
<z  a 
>-  ui 

o  :> 

K  CC 


72 


computer  run  times  for  the  TSAR  simulation  by  treatment  for 
the  no-attack  case.  Each  run  of  TSAR  averaged  3.73  minutes 
on  a  Gould  NP-1  supercomputer  for  a  total  of  477.37  minutes 
for  the  entire  no-attack  case. 

The  Attack  Case .  The  success  of  the  variance  reduction 
technique  is  more  apparent  in  this  case,  ranging  from  a  low 
of  18.92%  on  Day  8  to  a  high  of  81.90%  on  Day  2.  The  average 
percent  reduction  in  variance  is  38.81%.  Here  we  see  that 
the  blocking  based  on  the  seeds  for  the  TSARINA  attacks,  as 
well  as  on  the  random  number  streams  for  the  logistics 
processes  in  TSAR,  is  much  more  effective  in  reducing  the 
variance.  Table  4.4  shows  the  computer  run  times  for  the 
TSAR  model  when  attacks  were  present.  The  runs  averaged  2.79 
minutes  or  a  total  of  357.02  minutes  on  a  Gould  NP-1 
supercomputer.  While  the  simulations  for  this  case  do  not 
take  as  much  computer  time  as  in  the  no-attack  case,  the  cost 
of  additional  replication  would  still  be  significant, 
especially  considering  the  magnitude  of  the  experiment  and 
the  computer  time  involved. 

Summary .  A  single  replication  of  the  experiment  costs 
over  13.9  hours  of  computer  time  for  the  TSAR  model  when  run 
on  a  Gould  NP-1  supercomputer.  Computer  times  for  the 
TSARINA  attacks  were  not  tracked  but  are  very  small  in 
comparison  to  the  TSAR  run  times  since  TSARINA  runs  much 
faster  and  only  eight  attacks  were  used  (one  for  each  block) . 
Based  on  the  magnitude  of  the  computer  time,  it  is 


73 


<M00'<-(M^',rcDrjy3a30CMai^-QD 

C^N-H(T'CDt\iir5NDa50'f'<-CMm(N 

(s.-HO,fl'ryjinvflCD^flf^,<‘CDN'^,nin 

<  (\J  (\J  *— <  (\j  i— <  »"■<  r— <  f\J  (\J  *— *  *— < 


lD(\fOOD(\JOJ<J3(\l^lfi(\l(\l''fC\IC\lf\I 

\DOD-<C\J-«NVlfl(D-*C>JOCOCD-<in 

■-|0'in,<-^'(\in,)’ffnrnmQDrn'<,o^o 

<£~i-cyi'TtfN&0'J)(r>Ni£\S)incn 

i-<  ,-(  — .  —I  _l  —<  CM  ■"•  »-'•-< 


<00(D0f(i<00(Mt^tf0(M 

acr>fN.vflt\jcDfnacr'CDCT'mo'T<Nj^ 

in  on  rf  rn  co  n  f\i  cd"  o  (M  N-'  od  (M  n  rn  CD 
N-rmo^coo'OuooioincDa5inN 
•  — <  — <  c\j  •  •— <  •— <  •  •- *  r\i  •  •— <  •— * 


OIMlD^O'WriJCDOa^ffllOOOJN 

cKm-^idfNi-^rOrJV  in  o  d  in  o  cm  i/> 

inr^^inNm'vcr'iflCMNOoinorvjin 


O^'^lfltM’tOvflvfl^'QCOiD'^ODUJ 
cncDoa(\icDin-«a3iDO^^ONvDm 
. . 

innNtHtfOQ^^co-iinfMQcn 


vDf\J'*,',rfM(\Jvfl\fl<CDO\DCOOODO 
mvD'4-QoaocD(Na'(\i'<rvo\fivflcri(7'-' 
•  *(*«•!•••*••••* 
l/lNfMOO^xON'BAI^ttfON 
(M-.T'rNino'rao'»fOvJ5-'N’<-cr> 


-HNTTOCn'fOfMHUKIiai"'-''-' 

rvpg-<mi0CDCDiDiniflQ3O(rt->cnin 

<r>invj0cvm(^N--<»-i^f\ja3CDNin'r 


TlOOOlflO'flvSt'TNNOiJJvfiOJ 

■«-<\jmcr'<NQ0'C0N'*  mo'^inmin 

•£CDff'NTrfTNO'MflN^C"llN 


74 


undesirable  to  needlessly  replicate  the  experiment.  Thus  the 
variance  reduction  results  for  this  large-scale  simulation 
model  and  problem  are  beneficial. 

The  Daily  Metamodels 

Another  research  objective  is  the  estimation  of  simpler 

metamodels  from  the  simular  responses.  Before  deriving  the 

daily  metamodels,  we  test  whether  sorties  are  normally 

distributed  for  each  case.  Then  we  compute  some  sample  and 

population  statistics  for  selected  design  points  and  days. 

Next,  using  regression,  we  derive  a  metamodel  for  each  day  in 

each  case  (no-attack  and  attack) .  Each  is  a  reduced  model 

with  all  factors  (except  the  blocking  factors)  sign: ficant  at 

the  0.10  level;  models  are  presented  by  case.  For  each 

model,  we  examine  the  coefficient  of  multiple  determination 
2 

(R  )  and  the  residuals  to  determine  the  aptness  of  the  model. 
Interpretations  of  the  models  over  the  entire  thirty-day  time 
period  are  reserved  for  Chapter  V. 

Coding  of  the  Design  Matrix 

3efore  discussing  the  results  of  the  analyses  in  this 
research,  we  must  first  be  aware  of  the  method  used  to  code 
the  indicator  variables  associated  with  the  resource/policy 
variables  and  the  blocks.  Resource  variables  are  coded  such 
that 

Xj.j  =  1  if  resource  j  is  at  the  high  level  on  Day  1, 

Xij  =  0  if  resource  j  is  at  the  low  level  on  Day  1, 
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for  j  =  1, 2,  .  .  .  , 10  . 

Block  va-iables  are  coded  such  that 

Bfc  =  1  if  observation  is  from  block  k 
=  -1  if  observation  is  from  block  8 
=  0  otherwise 
for  k  =  1,2,  .  .  .,7. 

This  coding  scheme  yields  sound  results  in  our  comparative 
analyses  where  the  intercept  term  reflects  the  case  where  all 
resource/poiicy  variables  are  at  the  low  level .  The  beta 
coefficients  then  reflect  the  change  in  sorties  flown  when 
the  factor  or  interaction  term  moves  from  the  low  level  to 
the  high  level.  However,  with  our  coding  scheme,  the  beta 
coefficients  are  not  the  exact  effect  of  the  factor's 
resource  level  and  a  transformation  is  needed  to  isolate  the 
effect.  These  transformations  are  possible  conceptually,  but 
computationally  they  are  ver’’  involved  and  complicate!.  The 
exact  effect  of  each  resource/policy  variable  can  be  computed 
by  using  the  following  coding  scheme: 

Xij  =  1  if  resource  j  is  at  the  high  level  on  Day  1, 

=  -1  if  resource  i  is  at  the  low  level  .m  Day  1, 
for  j  =  1 , 2, . . . , 10 . 

Appendix  A  shows  the  equivalency  of  the  two  model  f.  .ms . 

Since  the  analysis  results  are  not  affected,  we  leave 
the  ceding  as  is  for  this  research.  Future  analyse"  will 
incorporate  a  consistent  coding  scheme  for  the  resources 
usinc  (1,-1)  in  place  of  (1,0)  respectively  to  denote  high 
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and  low  resource  levels  if  one  desires  the  beta  coefficients 
to  reflect  the  exact  effect  of  each  factor  or  interaction. 

Test  for  Normality 

The  Shapiro-Wilk  test  is  used  to  determine  whether  daily 
sorties  are  likely  to  be  normally  distributed.  We  compute  6 
Shapiro-Wilk  test  statistics  for  each  case  (i.e.,  attack  and 
no-attack)  based  on  20  independent  replications.  Within  each 
case,  we  test  for  normality  on  Days  1,  5,  and  30  for  design 
point  (a)  where  all  factors  are  at  the  high  level,  and  design 
point  (b)  where  all  factors  are  at  the  low  level.  Sample 
statistics  and  the  results  of  the  normality  tests  are  shown 
in  Table  4.5. 

The  Shapiro-Wilk  test  results  indicate  that,  for  the 
most  part,  daily  sortie  counts  appear  to  be  normally 
distributed  based  on  the  six  selected  design  points  for  each 
case.  Three  exceptions  are  noted  in  Table  4.5  for  which 
standard  normal  quantile  plots  are  shown  in  Figures  4.1  - 
4.3.  With  the  exception  of  Day  1  of  the  attack  case  when  all 
factors  are  low,  there  are  no  major  departures  from  the 
assumption  of  normality.  Additional  Shapiro-Wilk  tests  were 
conducted  for  Days  2-4  of  the  attack  case  with  all  factors 
low  to  see  if  sortie  counts  for  these  days  also  appeared  to 
be  nonnormal.  The  results  indicated  that  we  would  accept  the 
null  hypothesis  of  normal  distribution  of  sortie  counts. 

Given  the  overall  results  we  conclude  that  daily  sortie 
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Table  4.5 

Sample  Statistics  and  Shapiro-Wilk  Test  Results 


NO-ATTACK  CASE 


All 
Day  1 

Factors 
Day  5 

High 

Day  30 

All  Factors 
Day  1  Day  5 

Low 

Day  30 

Sample 

Mean 

266.3 

215.4 

70.0 

262.0 

183.4 

55.6 

Sample 

Variance 

59.99 

107.31 

182.84 

59.26 

267.94 

117.00 

Shapiro-Wilk 
Statistic  W 

0.9553 

0.8376 

0.9576 

0.9693 

0.9625 

0.9448 

Signif icance 
Probability 

0.4636 

0.0027 

0.5043 

0.7323 

0.5974 

0.3066 

Ho:  Normal 

Accept 

Reject 

Accept 

Accept 

Accept 

Accept 

ATTACK 

CASE 

All 
Day  1 

Factors 
Day  5 

High 

Day  30 

All  Factors  Low 
Day  1  Day  5  Day  30 

Sample 

Mean 

112.0 

testfcsss 

127.0 

50.1 

87.4 

89.0 

33.5 

Sample 

Variance 

711.73 

701.52 

372.79 

763.31 

1138.47 

213.10 

Shapiro-Wilk 
Statistic  W 

0.9561 

0.8643 

0.9851 

0.7403 

0.9746 

0.9340 

Significance 

0.4770 

0.0085 

0.9710 

0.0001 

0.8335 

0.1948 

Probability 


Ho:  Normal  Accept  Reject  Accept 


Reject  Accept  Accept 
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440 
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counts  for  both  cases  appear  to  follow  the  normal 
distribution . 

Possibly  contributing  to  nonnormality  for  the  attack 
case  are  four  treatments  where  the  air  base  never  recovers 
from  the  attacks  during  Days  1-5.  These  are  unlikely 
situations  since  it  is  more  probable  that  human  ingenuity 
would  find  ways  to  circumvent  almost  any  problem  to  restore 
flying  operations  to  some  minimum  level  during  the  thirty-day 
period.  These  four  cases  appear  to  be  a  failure  of  TSAR  to 
model  this  type  of  situation  rather  than  probable 
occurrences.  However,  the  cases  were  left  in  the  analysis  as 
representative  of  worst  case  situations. 

Statistics  For  Design  Point  k 

Here  we  focus  on  our  two  selected  design  points,  (a)  all 
factors  high  and  (b)  all  factors  low. 

Confidence  intervals  for  |i.^(i),  rhe  mean  number  of 

sorties  flown  on  Day  i  =*  1,  5,  30  at  design  point  k=a,b  for 

★ 

the  no-attack  case,  and  the  same  measure,  n^(i),  for  the 
attack  case,  are  shown  in  Tables  4.6  and  4.7,  respectively. 
These  95%  confidence  intervals  are  based  on  equations  (3.7) 
and  (3.8)  . 

For  the  most  part,  the  widths  of  these  95%  confidence 
intervals  for  the  mean  response  are  reasonably  narrow.  All 
of  the  sample  means  (except  one)  from  our  20  independent 
replications  fall  within  their  respective  confidence 


Confidence  Intervals  for  }lk.(i)  in  the  No-Attack  Case 
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interval.  The  one  exception  (Day  1,  all  factors  high,  no¬ 
attack  case)  exceeds  the  upper  limit  by  less  than  1  sortie. 

Figure  4.4  compares  the  confidence  intervals  given  in 
Tables  4.6  and  4.7.  On  Day  1  we  see  that  there  appears  to  be 
definite  differences  between  the  attack  and  no-attack  cases, 
although  the  confidence  intervals  do  overlap  within  each  case 
for  design  point  (a)  and  (b) .  By  Day  5  the  confidence 
intervals  are  wider  for  all  four  case-design  point 
combinations,  but  have  moved  closer  to  each  other  with  some 
overlapping.  By  Day  30,  the  confidence  intervals  are  nearly 
the  same,  indicating  that  previous  differences  tend  to 
balance  out  in  the  long  run. 

Homogeneity  of  variance  test  results  are  shown  in  Table 
4.8.  The  results  for  the  within-case  variance  all  indicate 
that  the.  null  hypothesis  of  ecu3l  variances  at  design  points 
for  Day  i  should  not  be  rejected.  This  is  important  since 
many  (128)  design  points  are  used  to  estimate  each  daily 
metamodel  and  homoscedasticity  is  a  requirement  for  minimum 
variance  unbiased  estimators  using  ordinary  least  squares 
procedures  (Neter  et  al . ,  1985).  In  contrast.  Table  4.8 
results  for  between-cases  variance  indicate  that  we  should 
reject  the  null  hypothesis  and  accept  the  alternate 
hypothesis  of  unequal  variances  between  the  no-attack  and 
attack  cases  for  the  same  day  and  design  point.  This 
supports  the  experimental  design  decision  to  keep  the  two 
cases  separate.  One  exception  to  note  is  that  we  do  not 


Results  for  Homogeneity  of  Variance  Tests 
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nonequal  variance 
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reject  the  null  hypothesis  for  Day  30.  It  seems  that  the 
differential  effect  of  the  attacks  has  dissipated  by  then. 
Overall,  then  it  appears  that  variance  is  homogeneous  within 
cases,  but  heterogeneous  between  cases. 

Summary .  As  a  result  of  the  normality  and  variance 
tests,  we  conclude  that  the  data  contain  no  major  departures 
from  the  assumptions  of  normality  and  homoscedasticity .  The 
differences  in  variance  between  the  two  cases  necessitate  the 
use  of  special  procedures  as  presented  in  Chapter  III  to 
allow  comparisons  of  the  attack  and  no-attack  cases. 

Me.t-gmodel.s_f.Qr  the  No-Attack  Case 

Table  4.9  summarizes  the  estimated  daily  metamodels  for 
the  no-attack  case.  The  beta  coefficients  for  each  model  are 
shown  only  when  that  rriain  effect  or  interaction  term  is 
significant.  The  beta  coefficient  represents  the  change  in 
number  of  sorties  flown  when  a  factor  is  at  the  high  level  as 
opposed  to  the  low  level.  While  individual  beta  coefficients 
provide  some  insights  into  the  importance  of  a  main  effect, 
we  must  reserve  conclusions  until  the  effects  of  all  two-way 
interactions  have  also  been  included.  These  analyses  are 
found  in  Chapter  V  for  each  rriain  effect  and  its  interactions. 
Complete  regression  results  for  each  estimated  metamodel  are 
found  at  Appendix  A. 

Coefficients  of  multiple  determination  are  summarized  in 

2 

For  the  reduced  daily  models,  the  R  values 


Table  4.10. 


No-Attack  Case  —  Daily  Metamodels 
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range  from  .487  (Day  27)  to  .819  (Day  17)  with  an  average  of 

.679  over  all  thirty  models.  This  indicates  that  the 

metamodels  fail  to  capture  about  thirty  percent  of  the 

variance  even  with  blocking.  This  is  true  even  for  the  full 
2 

models  where  R  values  range  from  .609  (Day  4)  to  .871  (Day 
17) .  One  suspects  then  that  there  is  either  high  variability 
in  the  logistics  processes  modeled  by  TSAR  or  there  are  other 
significant  factors  which  are  omitted.  We  do  find  that  each 
daily  metamodel  is  highly  significant  as  indicated  by  the  F- 
statistics  in  Table  4.10.  Given  that  MS£(i)is  an  unbiased 

estimator  of  the  population  variance,  we  see  that  these 
values  appear  to  be  at  reasonable  levels. 

Residual  analyses  for  each  daily  metamodel  reveal  no 
radical  departures  from  the  assumption  that  the  error  terms 
are  normally  distributed  with  a  mean  of  zero.  Note:  This  is 
only  an  observation  since  we  cannot  legitimately  test  for 
normality  because  the  data  are  not  independent,  but  are 
correlated  because  we  used  common  random  numbers  within  the 
blocks.  Plots  of  the  residuals  versus  the  predicted  values 
reveal  no  major  problem  areas.  Test  results  and  complete 
data  on  the  residuals  are  found  at  Appendix  B. 

.Metamodels  for  the  Attack  Case 

Table  4.11  summarizes  the  daily  metamodels  for  the 
attack  case.  The  beta  coefficients  for  each  model  are  shown 
only  when  that  main  effect  or  interaction  term  is 


Attack  Case  —  Daily  Metamodels 
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significant.  The  beta  coefficient  represents  the  change  in 
number  of  sorties  flown  when  a  factor  is  at  the  high  level  as 
opposed  to  the  low  level.  While  individual  beta  coefficients 
provide  some  insights  into  the  importance  of  a  main  effect, 
we  must  reserve  conclusions  until  the  effects  of  all  two-way 
interactions  have  also  been  included.  These  analyses  are 
found  in  Chapter  V  for  each  main  effect  and  its  interactions. 
Complete  regression  results  for  each  estimated  metamodel  are 
found  at  Appendix  C. 

Coefficients  of  multiple  determination  are  summarized  in 

2 

Table  4.12.  For  the  reduced  daily  models,  the  R  values 

range  from  .468  (Day  6)  to  .927  (Day  2)  with  an  average  of 

.636  over  all  thirty  models.  These  R  values  tend  to  be 

better  during  the  period  of  attack  but  then  lower  than  those 

of  the  no-attack  case  on  other  days.  The  metamodels  fail  to 

capture  about  thirty-five  percent  of  the  variance  even  with 

2 

blocking.  This  is  true  even  for  the  full  models  where  R 

values  range  from  .591  (Day  6)  to  .942  (Day  2) .  One  suspects 

then  that  there  is  either  high  variability  in  the  logistics 

processes  modeled  by  TSAR  plus  that  of  the  attacks  or  there 

are  ocher  significant  factors  which  are  omitted.  Contributing 
2 

to  the  lower  R  values  here  are  the  four  cases  of  no  recovery 
which  tend  to  skew  the  results.  We  do  find  that  each  daily 
metamodel  is  highly  significant  as  indicated  by  the  F- 
statistics  in  Table  4.12.  Also,  the  population  variances,  as 
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estimated  by  MSe*(i)  and  MS^i),  appear  to  be  higher  than  in 
the  no-attack  case. 

Residual  analyses  for  each  daily  metamodel  reveal  no 
radical  departures  from  the  assumption  that  the  error  terms 
are  normally  distributed  with  a  mean  of  zero.  As  noted 
above,  the  test  for  normality  is  not  legitimate  since  the 
data  are  correlated.  Residuals  plotted  against  the  predicted 
values  reveal  no  major  problem  areas.  Test  results  are 
summarized  in  Table  4.10.  Complete  data  on  the  residuals  are 
found  at  Appendix  D. 

The  Impact  of  Attacks 

The  research  objective  here  is  to  determine  whether 
attacks  on  the  ax r  base  cause  a  significant  difference  in 
flying  performance.  Before  any  statistical  tests  are  done, 
an  informal  visual  comparison  of  the  daily  metamodels  for 
both  cases  (see  Tables  4.9  and  4.11)  reveals  that  each  day's 
models  for  the  two  cases  not  only  have  different  values  for 
the  beta  coefficients,  but  often  do  not  even  contain  the  same 
significant  factors  and/or  interaction  terms.  Thus  it 
appears  that  there  is  a  difference  in  the  two  cases  that  can 
be  attributed  to  the  attacks.  3elow  we  test  for  significant 
differences  in  the  mean  response  between  the  two  cases  and 
develop  confidence  intervals  for  the  differences. 

Since  sorties  flown  are  not  independently  and 
identically  distributed  at  each  design  point,  valid 


statistical  comparisons  or  tests  for  differences  must  be 
restricted  to  individual  design  points.  Differences  between 
the  two  cases  can  be  tested  as  developed  in  Chapter  III  based 
on  Student's  t-distribu-t ion .  From  (3.23)  and  (3.24)  we  have 
our  hypotheses  for  testing  the  difference  between  the  mean 

k 

number  of  sorties  flown  on  Day  i  with  attacks,  |l^(i),  and  the 
mean  number  of  sorties  flown  on  Day  i  without  attacks,  ^(i), 
at  a  design  point  k: 

Ho;  V1'  ■  tv1' 

Hj:  Hk(i>  <  Hk<i) 

These  hypotheses  are  tested  at  two  design  points,  (a)  all 
factors  at  the  high  level  and  (b)  all  factors  at  the  low 
level,  for  each  Day  i  =  1,  5,  30.  Test  results  are  shown  in 
Table  4.13. 

The  results  indicate  significant  differences  in  the  mean 
response  for  the  attack  and  no-attack  cases  at  5  of  the  6 
design  pcint-day  combinations  tested.  The  only  point  where 
no  difference  is  detected  is  Day  30  when  all  factors  are  at 
the  high  level.  It  appears  that  any  difference  caused  by  the 
attack  (as  indicated  by  the  results  for  Days  I  and  5)  have 
subsided  by  the  end  of  the  30-day  period  when  all  factors  are 
high.  As  noted  in  Chapter  III,  this  test  is  conservative  so 
that  the  5  cases  where  we  detect  significant  differences  at  a 

=  .05  are  really  significant  at  a  even  lower  level. 


Tests  for  Significant  Impact  of  Attacks 


Day  30  -39.6  -1.6525  -2.7364 

(214) 
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From  (3.33)  we  also  compute  a  conservative  95% 

* 

confidence  intervals  for  the  differences  ()lk(i)  “  Hk(i)) 
which  are  shown  in  Table  4.14.  These  confidence  intervals 
are  plotted  in  Figure  4.5  and  support  the  conclusions  drawn 
above  concerning  significant  differences  in  daily  sortie 
counts  attributable  to  attacks.  In  Figure  4.5  we  see  that 
the  confidence  interval  for  Day  30  differences  when  all 
factors  are  high  spans  zero.  This  leads  to  the  conclusion 
that  there  is  no  significant  difference  attributable  to 
attacks  in  this  case.  We  also  note  no  great  differences 
between  the  confidence  intervals  for  the  two  design  points 
suggesting  that  the  impact  of  attacks  overshadows  any 
difference  caused  by  resource  level. 

Overall,  we  conclude  that  attacks  do  make  an  impact  on 
the  number  of  daily  sorties  flown.  However,  this  impact 
seems  to  dissipate  as  time  passes.  These  results  are 
consistent  with  what  we  would  expect  given  the  destruction 
and  disruption  caused  by  attacks  on  an  air  base. 


for  Differences  Attributed  to  Attacks 
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CHAPTER  V  -  INTERPRETATION  OF  THE  METAMODELS 

This  chapter  addresses  the  estimated  metamodels  and 
compares  results  for  the  two  cases,  attack  and  no-attack. 
First  we  discuss  overall  general  results  for  the  no-attack 
case  as  compared  to  the  attack  case.  Next  we  focus  on  the 
significant  factors  and  interactions  for  the  no-attack  case. 
Similarly,  the  attack  case  is  then  presented.  Lastly  we  look 
at  each  factor's  contribution  to  sortie  generation  with  and 
without  air  base  attack. 

Overall  Comparison  of  the  No-Attack 

and . Attack  Cases 

One  would  expect  attacks  on  an  air  base  to  make  a 
difference  in  the  number  of  sorties  flown.  3esides  damage  to 
runways  and  taxiways  which  prevent  aircraft  from  taking  off, 
resources  are  destroyed,  people  are  injured  and  killed, 
utilities  such  as  water  and  electricity  are  lost,  and  a 
generally  chaotic  atmosphere  endures  until  operations  are 
restored  to  normal.  3eiow  we  will  examine  various 
combinations  of  artack/no-attack  and  high/low  resource  levels 
in  order  to  better  understand  the  overall  influence  of 
attacks  on  the  number  of  expecoed  daily  sorties  flown. 


Ill 


Attack  Versus  No-Attack 

As  discussed  above,  the  experimental  design  was  applied 
to  an  attack  scenario  and  repeated  with  a  no-attack  scenario. 
Here  we  compare  overall  results  for  the  two  experiments 
focusing  on  the  effect  of  the  attack  on  similar  resource 
postures . 

All  Factors  Low.  As  can  be  seen  in  Figure  5.1,  when  all 
resource  levels  are  low,  more  sorties  are  flown  in  the 
absence  of  attacks  then  when  the  base  is  attacked.  The 
plotted  lines  are  based  on  the  estimated  daily  metamodels 
with  all  resources  at  the  low  level.  The  lines  are  nearly 
congruent  from  Day  6  on  with  an  approximate  difference  of  40- 
50  sorties  per  day  on  average.  During  the  attack  period 
(Days  1-5),  the  difference  between  the  two  lines  is 
especially  significant.  Figure  5.2  shows  the  expected 
differences  between  the  two  cases.  Thus  it  appears  that  the 
attacks  reduce  the  capability  of  the  logistics  infra¬ 
structure  to  produce  sorties  when  all  resources  are  at  the 
low  level . 

All  Factors  High.  When  all  factors  are  at  the  high 
level,  the  gap  between  the  attack  and  no-attack  cases  closes 
to  about  30  sorties  per  day  after  the  attacks  stop  on  Day  5 
(see  Figures  5.3  and  5.4).  Again  there  are  large  differences 
during  the  attack  period.  On  Day  27  the  lines  actually  cross 
with  the  attack  case  flying  more  sorties  per  day.  Whether 
this  trend  would  continue  after  Day  30  is  unknown. 


Sorties 


Sorties 
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All  Factors  Low 


NO  ATTACK  MINUS  ATTACK 
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Figure  5.2 

Difference  Between  Attack  and  No-Attack  Cases  with  All 

Factors  Low 
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All  Factors  High 
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260 
250 
240 
230 
220 
210 
200 
190 
180 
170 
160 
150 
140 
130 
120 
1  10 
100 
90 

)  80 
70 
60 
50 


Sorties 


115 


All  Factors  High 


NO  ATTACK  MINUS  ATTACK 


1  6  It  IS  21  26 

Doys 


Figure  5.4 

Difference  Between  Attack,  and  No-Attack  Cases  with  All 

Factors  High 


116 


Generally,  when  we  have  high  factor  levels  without  attack, 
many  more  sorties  are  flown  which  also  means  that  more 
aircraft  are  lost  (based  on  a  rate  per  sortie  flown)  and 
expendable  resources  are  consumed  faster.  As  a  result  fewer 
aircraft  are  available  to  fly  in  the  later  days  as  compared 
to  the  attack  case  where  fewer  sorties  were  flown  initially 
(and  therefore  fewer  losses) .  From  the  resource  consumption 
point  of  view,  fewer  sorties  can  be  supported  during  the 
later  days  because  of  the  depletion  of  consumable  stocks  such 
as  missiles  and  fuel.  With  attacks,  the  resource  pools  do 
not  draw  down  as  4'uickly  because  the  attacks  delay  flying  and 
hence  consumption  of  resources. 

ail._Fgct.oxs.  High  Versus  All  Factors.  Low 

Here  we  examine  the  influence  of  resource  postures 
separately  by  scenario. 

No-Attack  Case.  Figure  5.5  depicts  the  comparison  of 
the  different  resource  postures  over  time  in  the  absence  of 
attacks.  Here  we  see  a  definite  difference  on  Day  2  which 
remains  until  Day  27.  The  differences  during  the  attack 
period,  however,  are  not  as  great  as  those  seen  above  when 
similar  resource  levels  were  compared  with  attacks  as  the 
variable  effect.  The  lines  actually  cross  on  Day  27  and  the 
"all  low  factor"  case  begins  to  fly  more  sorties  per  day, 
albeit  small  differences.  Figure  5.6  portrays  the  levels  of 


the  daily  differences. 
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Figure  5.5 

No-Attack  Case  —  All  Factors  High  Versus  All  Factors  Low 


All  High  versus  All  Low 

NO  ATTACK  CASE 
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Ail  Factors  High 


INCREASE  FROM  LOW  TO  HIGH 
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Figure  5.6 

No-Attack  Case  —  Difference  Between  All  Factors  High 

and  All  Factors  Low 
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With  Attack  Case.  In  Figure  5.7  we  see  that  the  two 
postures  yield  different,  but  nearly  congruent  results  when 
attacks  are  present.  Differences  in  expected  daily  sorties 
flown  are  shown  in  Figure  5.8.  It  appears  that  the  attacks 
have  a  similar  effect  on  sorties  flown  and  the  factor  levels 
determine  the  level  of  sortie  generation  attained. 

Nummary..; - Ml  .Factors  High  .  Versus  All  Factors  .Low  and  Attack 

Versus  ..No-Attack 

Figure  5.9  depicts  the  four  cases  discussed  above.  High 
factor  levels  appear  to  outperform  low  factor  levels  given 
similar  attack  conditions.  Results  for  the  "all  high  - 
attack"  case  are  actually  better  than  the  "all  low  -  attack" 
case  after  the  attacks  stop  on  Day  5.  Thus  it  appears  that 
high  resource  levels  are  desirable  whether  or  not  attacks  are 
expected.  From  Day  6  on,  all  the  lines  are  fairly  congruent 
and  decline  linearly  in  the  number  of  sorties  flown  per  day. 
The  "all  low  -  attack"  case  seems  to  provide  a  lower  bound 
while  the  "all  high  -  no-attack"  case,  for  the  most  part, 
provides  an  upper  bound.  The  two  remaining  cases,  "all  low  - 
^o-attack"  and  "all  high  -  attack"  are  very  similar  after  the 
attack  period. 

Nor Attack  Case  Factor  Results 

Overall  results  for  the  no-attack  case  are  shown  in 
Table  4.8  which  depicts  the  daily  metamodels  found  by 
stepwise  regression.  Entries  in  the  table  are  the  beta 


Figure  5.7 
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Figure  5.8 

Attack  Case  —  Difference  Between  All  Factors  High  and 

Ail  Factors  Low 
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Figure  5.9 

Comparison  of  Four  Resource-Attack  Combinations 
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coefficients  of  the  factors  that  were  significant  in  each 
daily  model.  The  beta  coefficient  represents  the  change  in 
number  of  sorties  flown  when  a  factor  is  at  the  high  level  as 
opposed  to  the  low  level. 

Table  4.8  also  provides  a  visual  picture  of  which 
factors  tend  to  be  significant  and  when.  For  example, 
looking  at  main  effects  only,  we  see  Filler  or  Replacement 
Aircraft  are  significant  factors  in  many  of  the  daily 
metamodels.  Other  main  factors  which  appear  to  be 
significant  over  time  are  Personnel,  Spare  Parts,  Missiles, 
Fuel,  and  ABDR.  The  remaining  main  effects  are  significant 
at  various  times,  but  appear  more  sporadically.  Figures  5.10 
and  5.11  depict  the  beta  coefficients  of  these  main  effects 
when  they  are  significant  factors  in  the  daily  metamodels. 

As  seen  in  the  figures,  some  contributions  are  positive  while 
others  are  negative.  Below  we  examine  those  main  effects 
which  tend  to  have  significant  beta  coefficients  over  time 
along  with  the  most  significant  two-way  interactions. 

Main  Effect :  Filler  or  Replacement  Aircraft 

As  seen  in  Figure  5.10,  Fillers  contribute  positively  to 
sorties  flown  in  most  of  the  daily  metamodels.  This  is 
especially  true  in  Days  10-26  where  additional  sorties  per 
day  are  as  high  as  44.  However,  negative  coefficients  are 
found  in  the  models  for  Days  28-30.  These  could  represent 
the  cost  of  flying  earlier,  perhaps  an  added  strain  on  the 
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resources  available  resulting  in  quicker  resource  consumption 
and  finally  inability  to  support  these  added  aircraft  in  the 
later  days.  Fillers  also  appear  to  have  important  two-way 
interactions  with  several  other  main  effects:  ABDR,  Spares, 
Fuel,  and  Missiles. 

Filler  x  ABDR  interactions.,  shown  in  Figure  5.12,  are 
mostly  positive  and  significant  after  the  second  week.  It 
appears  that  increased  ABDR  capability  allows  damaged 
aircraft  to  be  returned  to  flying  sooner  and,  along  with 
extra  aircraft  available  as  fillers,  the  result  is  more 
sorties  flown.  Note  that  the  coefficients  are  negative  for 
Days  28  and  29.  This  may  be  a  "cost"  resulting  from  flying 
in  earlier  days.  Figure  2.2  is  helpful  for  understanding 
this  non-intuitive  result.  Increased  ABDR  and  Fillers  lead 
to  more  aircraft  available  and  more  sorties  can  be  flown. 
However,  more  sorties  flown  also  means  more  aircraft  losses 
to  attrition;  thus  over  time  the  benefits  are  offset  and  can 
result  in  less  sorties  flown  in  later  time  periods. 

Filler  x  Scare  Parts  interactions,  are  shown  in  Figure 
5.12  and  generally  follow  the  same  pattern  as  the  Filler  x 
ABDR  interaction:  some  positive  coefficients  in  the  middle 
time  frame,  with  what  again  appears  to  be  a  cost  (in  the  form 
of  negative  coefficients)  in  the  last  five  days.  Apparently 
the  combination  of  fillers  (i.e.  more  aircraft)  plus  enough 
parts  to  keep  them  flying  results  in  more  aircraft  losses 
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earlier  and  fewer  aircraft  are  left  for  the  later  days  than 
if  those  additional  resources  had  not  been  available. 

Filler  x  Fuel  interactions,  shown  in  Figure  5.13,  have 
alternating  positive  and  negative  effects  in  the  first  and 
third  weeks,  and  then  large  positive  contributions  in  the 
last  four  days  —  over  40  sorties  per  day  for  the  last  three 
days.  The  occurrences  of  positive  coefficients  seem  to 
precede  occurrences  of  negative  coefficients,  suggesting  a 
small-scale  trade-off  of  flying  early  at  the  expense  of 
flying  the  next  day  or  two.  The  large  positive  contributions 
in  the  last  four  days  mean  that  the  extra  filler  aircraft 
lead  to  more  sorties  in  the  final  days  only  when  fuel 
resources  are  also  available  to  support  them. 

Fillers  x  Missiles  interactions,  shown  in  Figure  5.13, 
are  similar  to  the  Filler  x  Fuel  interactions  except  that  we 
see  notable  positive  contributions  to  flying  in  weeks  one  and 
two.  Apparently,  the  extra  missiles  allow  more  of  the 
additional  aircraft  to  fly  needed  sorties;  without  missiles, 
the  aircraft  will  not  be  sent  into  combat. 

Net  Effect  and  Summary.  Figure  5.14  shows  that  the  net 
effect  of  having  the  high  level  of  Fillers,  as  opposed  to  the 
low  level,  is  very  positive.  Figure  5.14  is  derived  by  using 
the  estimated  daily  metamodels  to  predict  the  differences  in 
sorties  flown  when  Fillers  are  at  the  low  level  and  all  other 
factors  are  high  as  compared  to  sorties  flown  when  all 
factors  are  high.  Interactions  are  also  taken  into  account 
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and  the  corresponding  coefficients  for  Filler  interactions 
drop  out  when  Fillers  are  low.  Overall,  Fillers  or 
replacement  aircraft  are  beneficial,  but  we  also  need  to  be 
able  to  support  them  with  missiles,  spares,  and  fuel  to 
realize  their  full  sortie  potential.  Also,  early  additional 
flying  may  result  in  negative  coefficients  in  later 
metamodels.  This  may  be  acceptable  when  early  sorties  might 
have  more  bearing  on  the  outcome  of  the  war  than  later 
sorties . 

Main  Effect;  ■■.Personnel 

The  contributions  of  the  Personnel  main  effect  are  shown 
in  Figure  5.10  along  with  the  other  important  main  effects. 
Personnel  show  significant  early  contributions  where  the 
demands  for  flying  are  highest.  Because  more  people  are 
available  to  fix  and  service  aircraft,  more  planes  are  ready 
to  fly  sooner.  But  we  also  see  a  negative  impact  after  Day 
17,  an  apparent  cost  resulting  from  the  early  benefits  of 
having  a  high  level  of  Personnel.  Referring  to  Figure  2.2, 
more  people  lead  to  more  flyable  aircraft  which  in  turn  lead 
to  more  sorties  flown.  More  sorties  flown  means  more 
resources  are  consumed  and/or  mere  aircraft  are  attrited  and 
thus  less  aircraft  are  available  to  fly  later.  Personnel 
also  appears  to  have  important  two-way  interactions  with 
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Personnel  x  Missiles  interactions,  shown  in  Figure  5.15, 
contribute  large  quantities  of  daily  sorties  (15  to  59)  to 
the  flying  effort  during  Days  7-10.  Then  Days  16-20  have 
negative  coefficients.  People  are  important  in  the  assembly 
of  delivered  missile  components.  Having  high  levels  of  both 
people  and  missiles  potentially  allows  more  aircraft  to  be 
armed  and  flown  with  missiles.  Negative  contributions  are 
most  likely  due  to  the  earlier  additional  flying  enabled  by 
the  Personnel  x  Missiles  interaction  where  additional 
aircraft  are  attrited  and  thus  not  available  later  as  they 
would  otherwise  have  been. 

Personnel  x  ABDR  interactions  show  steady  positive 
contributions,  mostly  in  the  middle  time  periods  (see  Figure 
5.15).  The  high  level  of  ABDR  allows  battle-damaged  aircraft 
to  begin  repair  sooner,  while  the  high  level  of  maintenance 
people  allows  the  aircraft  to  be  made  flyable  quicker.  Since 
the  nature  of  the  ABDR  process  is  generally  long,  the 
contributions  of  this  interaction  appear  later  in  the  time 
periods . 

Net  Effect  and  Summary.  Some  areas  in  the  logistics 
infrastructure  rely  heavily  on  manpower,  such  as  we  see  above 
with  missile  assembly  and  ABDR  processes.  Contributions  tend 
to  be  positive  when  high  personnel  levels  are  combined  with 
the  needed  resources  in  such  labor-intensive  processes. 

Figure  5.16  portrays  a  positive  net  effect  of  personnel  in 
the  first  18  days,  but  also  highlights  the  resulting  negative 
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Personnel  Interactions 
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□  personnel  +  personnel  x  missile  x  personnel  x  ABDR 


Figure  5.15 

No-Attack  Case  —  Personnel  Interactions 
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impact  felt  in  the  later  days.  There  are  several  sporadic 
two-way  interactions  with  personnel  that  have  negative 
coefficients  in  the  later  time  periods.  These  will  not  be 
discussed  since  the  pattern  over  time  is  not  significant,  but 
the  coefficients  can  be  seen  in  Table  4.8.  The  negatives  do 
offset  some  of  the  positive  gains  of  the  personnel 
interactions  in  the  first  18  days.  Again,  the  early  benefits 
may  outweigh  the  later  costs  if  the  early  flying  has  more 
bearing  on  the  outcome  of  the  war  than  the  later  sorties. 

Main  Effect; _ Sp.aj.£-B.axt..s. 

The  contribution  of  Spares  is  almost  entirely  positive 
and  found  in  the  second  week  (see  Figure  5.10).  Overall  this 
seems  to  indicate  that  the  low  level  of  Spares  is  sufficient 
for  the  first  week,  but  then  more  parts,  as  provided  by  the 
high  level,  would  increase  the  number  of  sorties  flown. 
Component  repair  backlogs  might  also  contribute  to  the  need 
in  the  second  week,  especially  since  Spares  do  not  remain 
significant  in  the  later  daily  metamodels.  Spares  do  have 
some  important  two-way  interactions  with  other  main  effects. 
The  most  notable  are  the  interactions  with  Missiles,  Support 
Equipment,  and  Fillers. 

Spares  x  Missiles  interact  ions  are  shown  in  Figure  5.17. 
For  the  most  part,  the  coefficients  are  negative  in  the 
middle  time  frame,  but  during  the  last  four  days  we  find 
positive  coefficients.  The  coefficients,  representing  the 
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Figure  5.17 

No-Attack  Case  —  Scare  Part  Interactions 
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daily  contribution  to  sorties  flown  when  both  Spares  and 
Missiles  are  at  high  levels,  are  fairly  small  in  magnitude. 
The  negative  contributions  range  from  -6.4  to  -15.2  sorties 
per  day  while  the  positive  range  from  8.7  to  15.3  sorties  per 
day.  The  negative  coefficients  could  result  from  earlier 
flying  since  increases  in  both  spares  and  missiles  lead  to 
greater  numbers  of  aircraft  available  to  fly  and  hence  more 
sorties  flown  and  aircraft  lost.  Increased  flying  also  means 
greater  resource  consumption  with  less  available  to  support 
later  flying  demands.  Thus  there  seems  to  be  a  cost  for  the 
early  sorties  as  discussed  above  for  other  interaction  terms; 
here,  however,  the  cost  (reflected  by  negative  contributions) 
appear  in  the  middle  rather  than  at  the  end  of  the  30-day 
period.  The  positive  contribution  at  the  end  might  be  a 
cyclic  effect  begun  by  the  earlier  flying.  Here  the  lost 
aircraft  mean  fewer  sorties  are  flown  in  the  middle  periods, 
and  this,  in  turn,  means  less  resources  are  consumed.  As  a 
result  there  are  more  resources  per  aircraft  remaining  and 
more  sorties  can  be  supported.  The  process  appears  to  be 
cyclic  with  costs  and  benefits  associated  with  different  time 
periods . 

Spares  x  Support  Equipment  interactions,  shown  in  Figure 
5.17,  are  mostly  positive  and  occur  predominantly  in  the 
middle  time  periods.  Support  equipment  are  necessary  for 
many  of  the  repairs  on  aircraft  and  their  components.  A  high 
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level  potentially  allows  more  aircraft  to  be  worked  on  and 
thus  use  the  high  level  of  Spares  available. 

Soares  x  Filler  Aircraft  interactions  are  discussed 
above.  Figure  5.17  shows  the  interaction  coefficients  along 
with  the  Spares  main  effect  and  other  important  Spares  two- 
way  interactions. 

Net  Effect  and  Summary.  The  availability  of  Spare 
Parts,  high  level  versus  low  level,  has  both  positive  and 
negative  effects.  Figure  5.17  shows  this  clearly.  From 
Figure  5.18,  the  net  effect  is  cyclic,  but  mostly  positive, 
especially  in  the  middle  time  periods.  There  are  notable 
negative  contributions  in  the  last  five  days  which  appear  to 
be  the  cost  of  earlier  flying  allowed  by  the  high  resource 
levels . 

Main  Effect ; _ Missiles 

Missiles,  as  seen  in  Figure  5.10,  appears  to  add  sorties 
early  in  the  flying  effort,  but  then  costs  sorties  later 
because  the  early  "extra"  flying  results  in  additional 
aircraft  losses  and/or  resource  consumption  and  hence  less 
flying  later.  Missiles  has  important  two-way  interactions 
with  many  of  the  other  main  effects. 

Missiles  x  A3DR  interactions  are  shown  in  Figure  5.19. 
Generally,  we  see  negative  contributions  in  the  first  14 
days.  High  levels  of  both  factors  ought  to  allow  increased 
sorties.  Increased  A3DR  capability  makes  more  aircraft 
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available  that  ordinarily  would  not  be  repaired,  while  the 
high  level  of  Missiles  allows  more  of  the  flyable  aircraft  to 
be  armed  and  fly  into  combat.  However,  the  early  flying, 
especially  in  the  first  few  days  where  attrition  rates  are 
highest,  may  increase  aircraft  losses  and  thus  decrease  the 
number  of  sorties  flown  later.  After  Day  22,  we  see  the 
expected  effect  where  both  resources  allow  more  aircraft  to 
be  ready  to  fly  and  we  have  positive  coefficients. 

Missiles  x  Personnel  .interactions  are  discussed  above. 
Figure  5.19  depicts  the  coefficients  for  this  interaction 
term  along  with  several  of  the  other  Missiles  two-way 
interactions . 

Missiles  -.aL-Spar.es  interactions  are  discussed  above. 
Figure  5.19  shows  this  interaction  term  along  with  other 
important  Missiles  two-way  interactions. 

Missiles  x  Fuel  interactions,  shown  in  Figure  5.20,  are 
mostly  positive  and  found  after  Day  15  when  both  resources 
are  at  the  high  level .  Both  resources  can  be  seen  as 
essential  consumable  items  necessary  for  the  aircraft  to  fly. 
Having  more  available  means  more  aircraft  are  available  to 
fly. 

Missiles  x  Fillers  interactions  are  discussed  above. 
Figure  5.20  depicts  the  coefficients  for  this  interaction 
term  along  with  several  of  the  other  Missiles  two-way 
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Missile  Interactions 

NO  ATTACK  CASE 


O  missiles  +  missile  x  filler  A  missile  x  fuel 

Figure  5.20 

No-Attack  Case  —  Missile  Interactions  (2) 
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Net  Effect  and  Summary.  Missiles  and  its  important  two- 
way  interactions  with  other  main  effects  have  a  mixed  overall 
impact.  From  Figures  5.19  and  5.20,  it  is  obvious  that  some 
effects  are  negative  while  others  are  positive  and  we  would 
expect  some  of  these  to  cancel  each  other  out.  Figure  5.21 
shows  the  net  effect  when  we  use  the  daily  metamodels  to 
predict  the  difference  in  sorties  when  Missiles  is  at  the  low 
level  (all  other  factors  high)  in  comparison  to  all  factors 
at  the  high  level.  We  see  some  early  positive  contributions, 
especially  Day  9  at  110  sorties.  In  the  middle  time  period, 
the  coefficients  are  mostly  negative  and  then  shift  to  mostly 
positive  later  on.  The  overall  impact  seems  to  be  positive 
with  fairly  minimal  costs  in  terms  of  lost  sorties  later. 

Ha  in  ...Effect : _ Enel 

Fuel  by  itself  appears  sporadically  as  a  significant 
factor  in  the  daily  metamodels.  Always  negative  when  it  is 
significant.  Fuel  appears  in  only  six  metamodels  (see  Figure 
5.11) .  Since  the  high  and  low  levels  are  the  same  until  Day 
15,  we  would  expect  no  difference  in  effects  until  then. 

Thus  the  appearance  of  Fuel  in  metamodels  before  Day  15  is 
spurious.  While  the  main  effect  is  not  very  prevalent, 
several  two-way  interactions  are  important:  Fuel  x  Fillers 
and  Fuel  x  Missiles. 

Fuel  x  Fillers  interactions  are  discussed  above.  Figure 
5.22  depicts  the  coefficients  for  this  interaction  term  along 
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with  that  of  the  main  effect  and  the  Fuel  x  Missiles 
interaction . 

Fuel  x  Missiles  interactions  are  discussed  above.  Figure 
5.22  depicts  the  coefficients  for  this  interaction  term  along 
with  that  of  the  main  effect  and  the  Fuel  x  Fillers 
interaction . 

Net  Effect  and  Summary.  Other  two-way  interactions  with 
Fuel  occur  randomly  as  can  be  seen  in  Table  4.8.  Examining 
the  Fuel  main  effect  without  considering  its  two-way 
interactions  with  other  main  effects  results  in  the  erroneous 
conclusion  that  more  fuel  results  in  fewer  sorties  flown.  We 
must  look  at  the  net  effect  of  Fuel  based  on  the  metamodels 
containing  all  the  significant  effects.  The  net  effect, 
shown  in  Figure  5.23,  is  small  positive  and  negative 
contributions  with  no  discernible  pattern  through  Day  17 . 

Then  there  is  a  five  day  period  (Days  21  to  25)  with  negative 
coefficients  bordered  on  each  side  by  large  positive 
contributions.  Overall  it  appears  that  the  effects  of  Fuel 
and  its  two-way  interactions  with  other  main  effects 
essentially  cancel  out  in  the  first  seventeen  days .  After 
that  the  effect  tends  to  be  positive  (except  for  the  five  day 
period  from  Day  21  to  Day  25) .  Thus  Fuel  appears  to  be  more 
important  toward  the  end  of  the  period. 
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Main  Effect:  ABDR 

ABDR  by  itself  showed  only  sporadic,  mostly  negative 
coefficients  when  significant  (see  Figure  5.11).  This  seems 
to  indicate  that  early  repairs  allow  early  flying  and  thus 
the  loss  of  aircraft  that  otherwise  would  be  available  to  fly 
later . 

Significant  interactions  with  ABDR,  however,  tend  to  be 
positive  and  appear  primarily  after  Day  14  (see  Figure  5.24)  . 
An  exception  is  the  interaction  with  Missiles  which  appears 
negatively  four  times  in  the  first  two  weeks,  but  then  shifts 
to  positive  results  after  Day  23. 

Net  Effect  and  Summary.  The  overall  effect  of  ABDR  and 
its  interactions  are  shown  in  Figure  5.25.  The  coefficients 
oscillate  between  positive  and  negative  and  are  relatively 
small  in  magnitude  (i.e.  less  than  15  sorties  per  day).  ABDR 
has  positive  benefits  when  other  resources  are  also 
available.  Due  to  long  repair  times,  these  benefits 
generally  show  up  in  later  periods. 

Other  Ma.it  Effects:  AIS.  Attrition.  Support  Equipment. 
and  Recovery 

The  remaining  main  effects  appear  only  occasionally . 
3elow  are  the  net  effects  of  each  as  predicted  from  the  daily 
metamcdels  by  calculating  the  difference  in  sorties  when  all 
factors  are  at  the  high  level  and  when  all  are  high  except 
the  factor  of  interest. 
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No-Attack  Case  —  ABDR  Interactions 
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'i  Net  Effect  of  AIS .  Figure  5.2  6  depicts  the 

contributions  of  AIS  at  the  high  level  as  opposed  to  the  low 
level.  We  see  that  the  early  contributions  are  nearly  all 
positive,  while  later  contributions  are  negative.  Relative 
to  other  factors  discussed  above,  the  magnitude  of  the  daily 
sorties  is  small  across  all  days,  ranging  from  +12  to  -12. 

Net  Effect  of  Attrition.  Figure  5.27  depicts  the 
contributions  of  Attrition  over  time.  For  the  most  part,  the 
high  (i,e.  most  favorable)  level  of  attrition  has  a  positive 
effect  compared  to  the  low  level.  Magnitudes  of  additional 
sorties  per  day  are  small,  under  ten  sorties  per  day  except 
for  two  days  with  around  15  sorties. 

Net  Effect  of  Support  Equipment.  The  additional  sorties 

)  realized  from  having  Support  Equipment  at  the  high  level  are 

shown  in  Figure  5.28.  Most  are  positive  and  in  the  middle 
time  periods.  There  is  a  definite  negative  effect  in  the 
last  four  days  as  well  as  in  the  first  two  days.  Again  the 
relative  magnitude  of  the  difference  in  sorties  is  small. 

Net  Effect  of  Recovery.  The  contributions  of  Recovery 
resources  are  depicted  in  Figure  5.29.  Both  positive  and 
negative  effects  are  seen  across  the  entire  period.  This  is 
difficult  to  explain  since  Recovery  resources  are  relevant 
only  when  the  base  is  attacked;  thus  the  results  for  this 
fact ui.  a:s  spurious. 
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Figure  5.27 

No-Attack  Case  —  Overall  Contribution  of  Attrition 
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Recovery  Resources 
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Figure  5.29 

No-Attack  Case  —  Overall  Contribution  of  Recovery  Resources 
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Attack  Case  Factor  Results 

Overall  results  for  the  attack  case  metamodels  are  shown 
in  Table  4.11.  Entries  in  the  table  are  the  beta 
coefficients  of  the  factors  that  are  significant  in  each 
daily  metamodel  estimated  by  stepwise  regression.  The  beta 
coefficient  represents  the  change  in  the  number  of  sorties 
flown  when  a  factor  is  at  the  high  level  as  opposed  to  the 
low  level . 

In  comparison  to  the  no-attack  case,  the  attack  case 
results  are  a  much  '’cleaner”  visually  in  the  table,  i.e. 
factors  tend  to  show  definite  trends  over  time  and  with 
little  changing  of  the  signs  of  the  coefficients.  All  main 
factors  show  a  significant  trend  or  pattern  with  the 
exception  of  AIS  and  Spare  Parts.  Although  AIS  is  never 
significant  by  itself,  it  is  nonetheless  important  because  of 
interactions  with  several  other  main  factors.  Spare  Parts 
shows  two  days  with  negative  coefficients  (Days  2  and  3)  and 
then  several  intermittent  days  with  positive  contributions. 

It  also  has  two-way  interactions  with  other  main  effects  that 
are  important . 

Figures  5.30  and  5.31  show  the  beta  coefficients  of  the 
most  significant  main  factors  over  time.  These  factors  are 
A3DR,  Recovery,  Personnel,  Support  Equipment,  Attrition, 
Fillers,  Missiles,  and  Fuel.  With  only  minor  exceptions,  all 
of  these  main  effects  contribute  positively  to  the  sortie 
generation  effort.  Referring  to  Figure  2.2,  we  see  that  is 
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Significant  Main  Factors 
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Attack  Case  —  Significant  Main  Factors  (1) 
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exactly  the  result  we  would  expect.  Some  factors  contribute 
throughout  the  30-day  time  period,  while  others  are  either 
early  or  late  contributors.  The  early  to  mid-period 
contributors  appear  to  be  Recovery,  Personnel,  Missiles,  and 
Fuel.  Attrition  is  a  mid-period  contributor,  while  mid  to 
late  contributors  appear  to  be  Fillers,  AEDR,  and  Support 
Equipment.  Below  we  will  examine  each  of  the  main  effects 
and  their  most  important  two-way  interactions. 

Main  Effect.;  _  Recovery  Resources 

Recovery  resources  consist  of  the  equipment  and 
personnel  necessary  to  repair  runways,  taxiways,  and 
facilities  damaged  during  attacks  on  the  air  base.  Figure 
5.30  depicts  the  significant  coefficients  for  Recovery  along 
with  some  of  the  other  main  effects.  With  this  factor  at  a 
high  level,  additional  sorties  can  be  flown,  but  only  in  the 
first  five  days  which  is  when  the  attacks  occur.  This 
indicates  that  runways  and  taxiways  are  cleared  of  rubble  and 
holes  in  the  pavement  are  repaired  so  that  flying  can  resume 
sooner  than  at  the  low  level  of  Recovery.  The  magnitude  of 
the  coefficients  is  large  —  over  81  sorties  on  Day  2. 
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of  Recovery  is  very  significant,  but  limited  to  a  very  short 
period  of  time.  However,  the  value  of  these  sorties  flown 
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Recovery  Resources 

INCREASE  FROM  LOW  TO  HIGH  LEVEL 


Figure  5.32 

Attack  Case  —  Overall  Contribution  of  Recovery  Resources 
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may  be  very  important  and  influence  the  outcome  of  the  war; 
thus  the  importance  of  this  factor  should  not  be  discounted. 

Main  Effect;  Personnel 

The  Personnel  factor  seems  to  begin  its  contribution 
where  Recovery  stops.  Days  4  to  12  are  where  the  high  level 
of  people  has  a  positive  impact  on  the  level  of  sorties  flown 
(see  Figure  5.30).  More  people  available  generally  allows 
more  aircraft  to  be  repaired  or  serviced  at  any  point  in 
time.  Thus  aircraft  should  be  returned  to  flying  sooner. 
Expected  additional  sorties  per  day  due  to  Personnel  range 
from  9  to  19  during  this  period.  Personnel  also  has  several 
important  two-way  interactions  with  other  main  effects: 
Support  Equipment,  ABDR,  and  Missiles. 

■Irngortent  Interactions.  Figure  5.33  shows  the 
coefficients  of  the  daily  metamodels  where  Personnel  and  its 
interaction  terms  are  significant.  Two  negative  interactions 
with  personnel  are  Support  Equipment  and  ABDR.  Both  of  these 
interactions  tend  to  return  aircraft  to  operational  flying 
statue  sooner  than  if  the  resources  were  at  low  levels .  As  a 
result,  more  aircraft  are  flown  earlier,  with  more  early 
losses,  which  results  in  fewer  aircraft  to  fly  later.  Again, 
there  is  an  apparent  trade-off  between  early  sorties  and 
later  sorties.  In  contrast,  the  Personnel  x  Missile 
interaction  is  positive  and  appears  in  daily  models  after  Day 
12,  ranging  from  13  to  20  sorties  per  day.  Missiles  are 
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Attack  Case  —  Personnel  Interactions 
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delivered  as  components  which  must  be  assembled  manually.  If 
we  have  both  high  levels  of  missiles  (i.e.  lots  of 
unassembled  components)  and  the  people  to  assemble  them,  we 
will  potentially  have  more  combat-armed  aircraft  to  fly. 

Net  Effect  and  Summary.  Figure  5.34  depicts  the  net 
effect  of  increasing  Personnel  from  the  low  level  to  the  high 
level.  Overall,  the  impact  of  Personnel  is  positive, 
particularly  in  the  first  16  days,  although  we  do  see  some 
relatively  small  negative  results  after  that.  When 
significant,  the  high  level  of  Personnel  contributes  about  11 
extra  sorties  per  day  when  positive  and  "costs"  about  3 
sorties  per  day  when  negative. 

Main  Effs.c.t.;  . .  Fuel 

Fuel,  as  shown  in  Figure  5.31,  is  an  early  positive 
contributor  to  the  flying  effort.  However,  these  results  for 
the  main  effect  seem  spurious  since  there  is  no  difference 
between  the  high  and  low  levels  until  Day  15.  For  some 
reason,  all  of  the  positive  contributions  associated  with 
Fuel  in  the  daily  metamodels  come  before  Day  14.  However, 
Fuel  also  has  important  interactions  with  many  of  the  other 
main  effects. 

Important  Interactions.  Figures  5.35  and  5.36  show  the 
many  opposing  interactions  involving  Fuel.  From  these 
figures,  we  see  that  positive  coefficients  result  from  Fuel's 
two-way  interactions  with  Support  Equipment,  Fillers,  and 
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AIS .  These  interactions  are  predominant  in  the  last  fifteen 
days.  Generally,  the  high  levels  result  in  aircraft  either 
being  repaired  or  serviced  that  otherwise  would  not  fly  and 
thus  more  sorties  are  flown.  In  contrast,  Fuel  x  Attrition, 
Fuel  x  Spares,  and  Fuel  x  Missiles  have  negative 
coefficients.  In  general,  they  allow  more  flying  early  and 
thus  a  cost  is  paid  in  terms  of  less  flying  later  because  of 
aircraft  losses. 

Net  Effect  and  Summary.  Figure  5.37  shows  the  net 
effect  of  the  Fuel  resource  and  its  interactions.  Negative 
contributions  dominate  the  first  13  days/  then  the  net 
contributions  turn  mostly  positive  (but  very  small  except  for 
Day  24) .  Overall  these  results  for  Fuel  highlight  the 
complex  interdependencies  found  in  the  sortie  generation 
process.  The  answers  or  reasons  are  not  always  clear  or 
intuitive . 

Main  Effect:  A3DR 

The  significant  coefficients  of  ABDR  are  shown  in  Figure 
5.30  along  with  those  of  other  main  effects.  More  A3DR 
capability  generally  means  that  more  battle  damaged  aircraft 
return  to  flying  status  and  hence  more  sorties  are  flown. 

Such  positive  contributions  are  seen  on  Days  2  and  3,  and 
then  intermittently  in  daily  models  after  Day  13.  The 
positive  contributions  of  ABDR  as  a  main  effect  are  modified 
by  its  two-way  interactions  with  Missiles  and  Personnel. 
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Important  Interactions.  The  ABDR  interactions  have 
mixed  effects  and  are  shown  in  Figure  5.38.  The  ABDR  x 
Missiles  interaction  is  generally  positive  (appearing  in  the 
models  of  weeks  2  and  3) .  The  exception  is  some  very  large 
negative  coefficients  in  the  models  for  Day  2  and  Day  3.  The 
size  of  these  negative  coefficients  (-47  and  -17)  plus  the 
slow  nature  of  the  ABDR  process  make  it  unlikely  that  early 
flying  (i.e.  early  losses)  resulted  in  lost  sorties  on  Days  2 
and  3.  We  suspect  some  other  interaction  with  attacks  in 
these  early  days  that  is  not  obvious.  The  positive 
coefficients  in  weeks  2  and  3  are  due  to  more  aircraft 
available  from  battle-damage  repair  that  also  have  missiles 
available  and  thus  more  sorties  are  flown.  In  contrast,  the 
ABDR  x  Personnel  interaction  is  mostly  negative  and  appears 
predominantly  in  the  last  week  or  so.  This  is  due  to  more 
people  available  to  fix  damaged  aircraft  and  hence  many  more 
are  returned  to  flying  sooner.  Subsequently  more  are  lost  to 
attrition  and  unavailable  for  flying  in  the  later  days. 

Net  Effect  and  Summary.  The  expected  net  effect  of  ABDR 
and  its  interactions  is  shown  in  Figure  5.39.  Here  we  see  a 
negative  effect  in  the  early  days,  mostly  positive 
contributions  in  the  middle,  followed  by  small  negative 
contributions  during  the  last  six  days.  Overall  there  are 
some  complex  interdependencies  which  are  unclear,  especially 
in  the  first  week. 
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Attack  Case  —  Overall  Contribution  of  ABDR 
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Ma-in  Effect; _ Attrition 

Attrition  appears  as  a  significant  factor  primarily  in 
the  second  week.  It  is  plotted  in  Figure  5.31  along  with 
some  of  the  other  significant  main  effects.  By  the  second 
week,  enough  sorties  have  been  flown  so  that  the  attrition 
rates  become  more  favorable .  The  delay  in  the  appearance  of 
the  Attrition  factor  is  most  likely  caused  by  the  attacks  on 
the  air  base  which  prevent  or  slow  down  flying.  The  delay  is 
evident  in  Figure  5.40  where  the  Attrition  effect  and  its 
interaction  with  Fuel  are  shown. 

Important  Interactions.  While  the  high  level  (i.e.  more 
favorable)  attrition  level  resulted  in  positive  coefficients 
over  time,  the  Attrition  x  Fuel  interaction  is  negative, 
mostly  ranging  from  -13  to  -22  in  the  second  and  third  weeks. 
Having  fuel  available  allowed  more  sorties  to  be  flown  and 
thus  more  aircraft  to  be  lost  (even  though  the  rate  was 
lower)  early  on  so  that  fewer  sorties  per  day  were  eventually 
flown . 

Net  Effect  and  Summary.  Figure  5.41  shows  the  expected 
net  effect  of  having  Attrition  at  the  high  or  more  favorable 
level.  The  result  is  not  what  we  would  expect.  As  is 
evident,  the  effects  are  almost  all  negative.  This  is  a  good 
example  of  the  complexity  and  interdependencies  within  the 
logistics  infrastructure  and  the  air  base  system, 
particularly  in  a  hostile  environment. 
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Attack  Case  —  Attrition  Interactions 


174 


Attack  Case  —  Overall  Contribution  of  A- 


rition 


175 


Main  Effect:  Filler  or  Replacement  Aircraft 

Fillers  are  shown  in  Figure  5.31  along  with  other 
significant  main  effects.  Fillers  have  negative  coefficients 
in  the  models  for  Day  2  and  3,  but  are  positive  after  Day  7. 
The  Day  2  and  3  results  may  be  that  the  fillers  allowed  more 
sorties  to  be  flown  in  the  first  few  days  when  attrition 
rates  were  high,  and  thus  resulted  in  fewer  aircraft  to  fly. 
However,  the  size  of  the  coefficients  {-23  and  -42  sorties) 
makes  this  unlikely.  Here  again  it  seems  there  is  an 
interaction  of  some  sort,  possibly  caused  by  the  attacks, 
which  is  not  readily  apparent.  Whatever  the  reason,  the 
impact  is  short-lived  and  shifts  to  positive  coefficients 
which  one  would  expect:  more  aircraft  should  equal  more 
sorties.  Two  interaction  terms  help  to  increase  the  positive 
contribution  of  Fillers. 

Important  Interactions.  Fillers  interacts  positively 
with  both  Spares  and  Fuel  as  shown  in  Figure  5.42.  The 
Filler  x  Spares  interaction  appears  first  in  Days  2-6  with 
coefficients  ranging  from  18  to  60  sorties  per  day,  and  then 
again  after  Day  11  with  coefficients  ranging  from  11  to  28 
sorties  per  day.  The  Filler  x  Fuel  interaction  is 
intermittently  positive,  ranging  from  8  to  22  sorties  per 
day.  It  appears  in  7  daily  models,  mostly  after  Day  15. 

Net  Effect  and  Summary.  Figure  5.43  «how«?  the  net 
effect  of  having  Fillers  at  the  high  level  rather  than  the 
low.  As  is  evident,  Fillers  make  significant  positive 
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contributions  to  sorties  flown,  averaging  about  30  additional 
sorties  per  day.  Fillers  appear  to  be  most  beneficial  when 
combined  with  the  resources  to  support  and  fly  them. 

Main  Effect.: _ Missiles 

Missiles  are  important  early  positive  contributors  as 
seen  in  Figure  5.31.  Their  availability  allows  more  flyable 
aircraft  to  be  launched  on  combat  missions.  However, 

Missiles  also  has  important  two-way  interactions  with  other 
main  effects  which  complicate  the  effect  on  sorties. 

Important  Interactions .  Figure  5.44  depicts  several 
interactions  with  Missiles.  The  Missile  x  Spares  interaction 
has  negative  coefficients  in  the  first  week  which  means  more 
sorties  were  launched  with  subsequent  losses  during  the 
higher  attrition  periods.  The  Missile  x  Fuel  interactions 
are  negative  and  very  predominant  after  Day  10,  ranging  from 
-9  to  -22  sorties  per  day.  This  appears  to  be  from  the  early 
flying,  but  is  offset  when  the  positive  Missile  x  Personnel 
interaction  is  also  present.  This  interaction  term  is 
discussed  above  as  is  the  Missile  x  ABDR  term. 

Net  Effect  and  Summary.  Overall  the  Missile  main  effect 
has  both  positive  and  negative  aspects  when  the  complex 
interactions  with  other  main  effects  are  also  considered. 
Figure  5.45  shows  the  overall  net  effect  is  positive  after 
Day  7,  however,  two  very  large  negative  contributions  are 
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Attack  Case  —  Missile  Interactions 
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found  in  the  first  week.  Again  there  appear  to  be  many 
complex  interdependencies. 

Main  Effect:  Support  Equipment _ L££L 

Support  Equipment  contributes  positively  in  almost  every 
daily  metamodel  after  Day  4  (see  Figure  5.30).  These 
equipment  are  necessary  to  repair  and  service  aircraft  for 
flight,  and  having  more  available  leads  to  more  sorties 
flown.  SE  does  interact  with  other  main  effects  with  some 
positive  and  some  negative  results. 

Important  Interactions.  The  important  SE  interactions 

are  shown  in  Figure  5.46.  The  SE  x  Fuel  interaction  is 
positive  since  more  fuel  trucks  are  available  in  the  high 
level  of  SE  which  means  the  high  levels  of  fuel  available  can 
be  transferred  to  flyable  aircraft.  This  then  leads  to  more 
sorties.  We  notice  too  that  almost  all  of  these  interactions 
are  not  significant  until  Day  15,  which  is  when  the  first 
difference  in  the  high  and  low  levels  of  Fuel  occurs.  In 
contrast,  SE  interactions  with  AIS  and  Personnel  generally 
are  negative  in  the  last  two  weeks.  The  high  levels  allow 
more  flying  early  at  the  expense  of  later  flying.  SE  x  AIS 
did  have  positive  contributions  for  three  days  in  the  first 
week;  this  probably  helped  result  in  negative  coefficients 
later . 

Net  Effect  and  Summary.  In  Figure  5.47,  we  see  that  the 
net  effect  of  SE  over  time  is  very  much  positive.  The 
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Attack  Case  —  Support  Equipment  Interactions 
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Figure  5.47 

Attack  Case  —  Overall  Contribution  of  Support  Equipment 
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benefits  appear  to  outweigh  the  negative  contributions  from 
some  of  the  interaction  terms.  These  results  are  what  we 
would  expect  from  this  resource.  A  delay  in  the 
contributions  is  also  evident  in  Figure  5.47,  suggesting  that 
the  attacks  may  destroy  SE,  thus  making  the  high  starting 
resource  level  important  to  the  sortie  generation  effort. 

Other  Main  Effects;  ..Spares  and  ATS 

Even  though  Spares  only  shows  up  as  significant  in  five 
daily  models,  it  interacts  significantly  with  several  other 
factors.  Similarly,  AIS  is  never  significant  in  any  daily 
metamodel,  but  it  too  has  important  interactions.  These 
interaction  terms  are  discussed  below. 

Spares  interactions  are  shown  in  Figure  5.48.  The 
interactions  with  Fillers  and  AIS  result  in  very  consistent 
positive  contributions  to  sorties  flown  through  much  of  the 
30-day  period.  These  generally  reflect  more  parts  and 
components  available  to  repair  aircraft  so  they  can  fly.  On 
the  other  hand.  Spares  interactions  with  Fuel  and  Missiles 
result  in  negative  coefficients.  These  offset  the  positive 
contributions  and  may  result  from  early  flying  at  the  expense 
of  later  sorties.  The  net  effect  of  having  a  high  level  of 
Spares  is  shown  in  Figure  5.49.  The  positive  contributions 
outweigh  the  negative  for  the  most  part.  Only  three  days 
have  negative  contributions  and  these  are  relatively  small. 
Some  early  positive  contributions  are  seen  in  the  first  three 
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Attack  Case  —  Spare  Fart  Interactions 
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days,  however  after  Day  11,  Spares  at  the  high  level  appear 
to  be  worth  about  15  sorties  per  day. 


ATS  interactions  are  shown  in  Figure  5.50.  Here  we  see 
positive  contributions  from  the  interactions  with  Fuel  and 
Spares.  These  occur  entirely  after  Day  10.  As  is  often  the 
case,  we  have  another  interaction  term  that  contributes 
negatively,  thus  partially  offsetting  the  positive 
interaction  terms.  The  AIS  x  SE  interaction  is  negative 
after  Day  16,  although  it  did  have  positive  coefficients  in 
three  models  during  the  first  week.  The  net  effect  of  having 
the  high  level  of  AIS  is  mostly  positive,  as  shown  in  Figure 
5.51.  Most  of  the  benefit  is  expected  after  Day  10  and 
averages  9-10  sorties  per  day.  Only  three  days  show  negative 
^  results  and  these  are  small  (all  less  than  4  sorties  per 

day)  . 

Kev  Resources  Over  Time 

The  fourth  research  objective  is  to  identify  key 
resources  and/or  interactions  over  a  thirty-day  time  period 
with  and  without  air  base  attack.  To  make  this  assessment  we 
first  compare  the  contributions  of  each  factor  separately  for 
the  attack  and  no-attack  cases.  Table  5.1  depicts  the  net 
additional  sorties  flown  over  thirty  days  when  each  factor  is 
at  the  high  level  as  compared  to  the  low  level  (with  all 
other  factors  at  the  high  level) .  These  net  totals  are  based 
on  the  metamodels  and  include  significant  two-way 
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Attack  Case  —  AIS  Interactions 
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interactions  of  the  factor  with  other  main  effects.  We 
examine  each  case  separately  and  the  comparative 
contributions  of  each  factor  in  order  to  determine  which  are 
the  most  important  over  the  thirty-day  period.  From  these 
comparisons  we  will  then  form  some  conclusions  as  to  the  most 
important  factors  over  time  when  the  probability  of  attack  is 
unknown . 

Factor  A: .  Attrition 

Attrition  seems  to  have  opposite  effects  on  sortie 
performance  depending  on  whether  or  not  the  air  base  is 
attacked  (see  Figure  5.52).  The  attack  case  has  a  very 
definite  negative  pattern,  while  the  no-attack  case  reveals  a 
generally  positive  effect.  In  general,  the  contribution  of 
Attrition  in  the  attack  case  is  positive  as  in  the  no-attack 
case,  but  its  interaction  with  Fuel  is  negative  and  hence  the 
overall  negative  contribution.  Further  evidence  of  the 
opposite  effects  is  seen  in  the  net  totals  for  the  two  cases 
in  Table  5.1:  -114.9  sorties  for  the  attack  case  compared  to 

+78.2  sorties  for  the  no-attack  case. 

Summary .  Attrition  is,  for  the  most  part,  an 
uncontrollable  environmental  factor.  Its  impact  on  sortie 
performance  ranks  in  the  middle  as  compared  to  the  other 
factors  in  Table  5.1.  It  appears  that  the  effect  of 
attrition  is  unpredictable,  or  at  least  unstable,  in  an 
uncertain  environment . 
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Figure  5.52 

Overall  Contribution  of  Attrition  —  Attack  Versus  No-Attack 


193 


i  Factor  B:  Filler  Aircraft 

Replacement  aircraft  have  the  same  positive  effect  for 
both  cases.  It  is  the  greatest  contributor  to  sortie 
generation  in  terms  of  net  sorties  (see  Table  5.1)  with  and 
without  attacks.  In  Figure  5.53,  the  effect  of  fillers 
occurs  earlier  in  the  attack  case  because  aircraft  destroyed 
in  the  attacks  are  being  replaced  in  addition  to  attrition 
losses.  The  contributions  are  fairly  large:  28-30  sorties 
per  day  on  average  over  the  thirty  days  regardless  of  attacks 
or  not . 

Summary .  Filler  aircraft  appear  to  be  by  far  the  most 
significant  contributors  to  sortie  performance. 

.  r_actQX-  Cj _ ABPR 

) 

ABDR  has  similar  results  for  both  cases  as  shown  in 
Figure  5.54.  Generally  the  effects  move  in  the  same 
direction  although  the  timing  is  sometimes  different  for  the 
two  cases.  Overall,  the  net  effects  for  both  cases  are 
small,  40  to  45  sorties  over  the  thirty-day  period. 

Summary .  Overall  the  difference  in  performance  between 
the  high  and  low  levels  is  small.  Relative  to  the  other 
factors  (see  Table  5.1)  the  high  level  of  A3DR  does  not 
appear  to  make  a  very  significant  contribution  to  sortie 
performance  beyond  that  provided  by  the  low  level  of 
capability . 
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Figure  5.53 

Overall  Contribution  of  Filler  Aircraft  —  Attack  Versus 
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Factor  D:  Recovery  Resources 

As  expected,  recovery  resources  are  only  important  when 
the  air  base  is  attacked  as  can  be  seen  in  Figure  5.55. 

Large  positive  contributions  are  made  during  the  period  of 
attacks  with  net  sorties  of  +147.2  expected  over  the  thirty 
days  (Table  5.1). 

Summary .  The  impact  of  additional  recovery  capabilities 
is  evident  during  the  attack  period.  A  central  issue 
concerns  the  value  of  these  additional  sorties.  The  early 
time  period  of  a  war  may  be  very  crucial  and  our  ability  to 
keen*  flying  during  this  time  may  decide  the  eventual  outcome 
of  the  war.  Thus,  although  the  recovery  factor  is  not 
significant  throughout  the  entire  period,  it  might  very  well 
be  one  of  the  most  significant  factors  if  early  sorties  are 
indeed  the  most  valuable  sorties. 

Factor  £; _ P,grs.Qh.n,el 

Personnel's  contribution  to  sortie  generation  generally 
follows  the  same  pattern  in  both  cases.  However,  as  seen  in 
Figure  5.55,  the  effects  in  the  absence  of  attacks  are  more 
pronounced.  In  the  early  part  of  the  thirty-day  period, 
there  are  high  sortie  demands  where  many  technicians  are 
needed  to  fix  and  service  aircraft.  In  the  attack  case,  the 
high  sortie  rates  demanded  cannot  be  met  because  runways  are 
damaged  and  closed.  Therefore  the  workloads  are  reduced  and 
the  contribution  of  additional  people  is  not  as  great.  Table 


Figure  5.55 

Overall  Contribution  of  Recovery  Resources  —  Attack  Versus 

No-Attack 
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Figure  5.56 


Overall  Contribution  of  Personnel 


Attack  Versus  No-Attack 
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5.1  shows  that  the  net  contribution  of  people  in  either  case 
is  not  very  great.  This  result  is  counterintuitive  and  is 
due  primarily  to  negative  contributions  in  the  last  half  of 
the  thirty-day  period  which  offset  the  earlier  gains  of 
having  a  high  level  of  Personnel. 

Summary .  People  appear  to  be  most  important  when  there 
are  no  other  restrictions  on  the  flying  effort.  Thus,  in  the 
absence  of  attacks  which  prevent  flying  to  a  large  extent, 
people  are  significant  contributors  when  at  the  high  level  as 
opposed  to  the  low  level.  We  also  note  that  there  appear  to 
be  significant  penalties  in  later  days  due  to  the  added 
sortie  generation  capability  in  the  early  days. 

Factor  F;  AIS  Test  Sets 

During  the  first  fifteen  days  in  Figure  5.57,  we 
generally  see  the  same  directions  for  contributions  to  the 
sortie  effort.  After  that,  we  see  opposite  effects,  with 
positive  contributions  when  attacks  are  present,  and  negative 
contributions  in  the  absence  of  attacks.  This  indicates  that 
the  effect  of  the  attacks  is  important.  Possibly  parts  are 
destroyed  by  the  attacks  which  increase  the  reliance  on  the 
AIS  to  repair  malfunctioning  components.  Ordinarily  a 
replacement  part  would  be  in  stock  and  used  to  repair  the 
aircraft,  thus  shortening  the  time  the  aircraft  is  unflyable. 
In  this  case,  since  fewer  parts  are  available  due  to  the 
attacks,  inoperable  avionic  components  are  removed  from  the 
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Figure  5.57 

Overall  Contribution  of  AIS  —  Attack  Versus  No-Attack 
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aircraft,  repaired  on  the  AIS,  and  returned  to  the  aircraft. 
Thus,  the  greater  the  AIS  repair  capability,  the  shorter  the 
downtime  for  the  aircraft.  A  second  possibility  is  that  the 
attacks  damage  the  AIS  itself.  Here  the  availability  of  two 
sets  (the  high  level)  increases  the  likelihood  that  some  AIS 
capability  will  be  present  even  if  damage  occurs.  Overall  in 
Table  5.1,  the  net  sorties  gained  from  AIS  is  fourth  when 
attacks  occur,  but  ranks  last  in  the  absence  of  attacks . 

Summary.  Overall,  AIS  and  its  interactions  account  for 
6-7  sorties  per  day  on  average  over  the  thirty-day  period 
when  attacks  are  present.  Thus  the  contribution  appears  to 
be  fairly  small  with  little  gained  by  having  two  sets  instead 
of  one,  especially  in  the  no-attack  case. 

Easton-G.; _ Support  Equipment 

While  fairly  unimportant  in  the  no-attack  case,  support 
equipment  is  the  second  highest  net  contributor  over  the 
thirty-day  period  when  attacks  are  present  (see  Table  5.1  and 
Figure  5.58).  This  suggests  that  support  equipment  are 
destroyed  in  the  attacks  which  has  a  significant  impact  on 
subsequent  repairs,  servicing,  and  flying.  This  impact 
averages  13  sorties  per  day  over  the  entire  in  the  attack 
case . 

Summary .  The  benefits  of  the  high  level  of  support 
equipment  is  evident  in  the  attack  case,  while  the  low  level 
appears  to  be  sufficient  in  the  no-attack  case.  Thus  it 
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seems  that  adequate  protection  of  support  equipment  is  very 
important.  Assured  protection  may  allow  a  lower  level  of 
equipment  to  suffice  rather  than  procuring  more. 

Factor  H:  Spare  Parts 

Figure  5.59  shows  that  the  attack  and  no-attack  cases 
have  generally  the  same  results  for  Spare  Parts  until  the 
last  five  days  of  the  thirty-day  period.  Spares,  high  level 
versus  low  level,  appear  to  be  more  important  in  the  attack 
case,  averaging  an  expected  +12  sorties  per  day  over  the 
entire  period.  The  higher  level  of  spares  means  a  greater 
likelihood  of  having  parts  available  for  repairs  despite 
losses  during  the  attacks.  In  terms  of  net  sortie 
contribution  over-  thirty  days,  spares  is  the  third  most 
important  resource  in  the  presence  of  attack  and  ranks  fourth 
when  there  are  no-attacks  (see  Table  5.1). 

Summary .  Spares  and  its  interactions  with  other  main 
effects  are  important  to  the  sortie  generation  effort,  with 
and  without  attacks.  Significant  gains  can  be  made  with  a 
high  level  of  spares  when  attacks  are  likely. 

Factor  J ;  Missiles 

Missiles  are  more  important  when  there  are  no-attacks  as 
is  seen  in  Figure  5.50.  Table  5.1  also  reflects  this,  with 
missiles  ranking  number  two  in  the  no-atrack  case  on  the 
basis  of  expected  net  contribution  to  sorties  over  the 
thirty-day  period.  Missiles  rank  only  eighth  when  attacks 
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Figure  5.60 

Overall  Contribution  of  Missiles  —  Attack  Versus  No-Attack 
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are  present.  It  appears  that  the  attacks  slow  down  flying, 
thus  reducing  the  demand  and  consumption  of  missiles.  When 
there  are  no-attacks,  the  consumption  of  missiles  is  high  due 
to  the  high  sortie  rates  being  flown.  We  should  note  the 
very  significant  "spikes"  on  Days  8-10  in  Figure  5.60  which 
occur  because  the  low  case  is  running  out  of  missiles;  the 
positive  contributions  subside  because  of  deliveries  received 
the  afternoon  of  Day  10.  The  no-attack  case  has  a  similar 
positive  trend  at  the  end  of  the  period  where  missiles  are 
running  out  due  to  high  flying  levels  and  missile 
consumption . 

Summary .  Missiles  appear  to  be  most  important  where 
high  sortie  rates  are  flown.  To  sustain  these  flying 
)  demands,  consumable  resources  such  as  missiles  must  be 

available . 

Factor  K:  Fuel 

The  effects  of  Fuel  on  sortie  generation  are  generally 
opposite  for  the  attack  and  no-attack  cases  (see  Figure 
5.61) .  From  Table  5.1,  we  see  that  fuel  ranks  last  in 
importance  in  the  attack  case,  but  is  third  in  the  no-attack 
case.  Fuel  is  another  resource  that  is  affected  by  high 
consumption  rates  found  in  the  no-attack  case.  As  seen  in 
Figure  5.61,  the  expected  positive  contributions  come  mainly 
late  in  the  period  when  a  high  number  of  sorties  have 
accumulated  thereby  consuming  Fuel  resources.  In  comparison. 
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the  attacks  prevent  and  slow  the  flying  effort  and  hence  the 
demand  for  fuel.  Attacks  do  not  appear  to  cause  any 
significant  losses  of  fuel  stored  on  the  air  base. 

Summary.  Fuel  is  another  consumable  resource  necessary 
to  sustain  flying  at  high  sortie  rates.  Generally,  the 
amount  available  in  this  research  is  sufficient  until  later 
periods  where  the  additional  deliveries  provided  by  the  high 
level  make  a  difference. 

Overall  Importance  of  Factors. 

To  determine  the  most  important  factors,  we  first 
examine  the  attack  and  no-attack  cases  separately.  Then  we 
judge  the  effects  of  the  factors  across  the  cases,  i.e., 
which  factors  are  most  important  given  that  we  do  not  know 
whether  or  not  the  air  base  will  be  attacked. 

Attack  .Qas.e 

In  Table  5.1  the  factors  are  ranked  according  to  the 
expected  net  additional  sorties  over  thirty  days  when  the 
resource  is  at  the  high  level  as  compared  to  the  low  level. 
The  top  five  contributors  are  graphed  in  Figure  5.62  to  show 
comparatively  which  factors  contribute  the  most  per  day  over 
time.  Similarly,  Figure  5.63  depicts  the  remaining  factors. 
From  Figure  5.62,  it  is  obvious  that  Filler  Aircraft  is  the 
dominant  factor  across  the  period.  Support  Equipment  is  the 
second  leading-  contributor  across  the  time  period,  followed 
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by  Spares  whose  contributions  are  most  obvious  and  consistent 
after  Day  11. 

Two  resources  appear  to  be  important  for  certain  periods 
of  time.  Recovery  resources  makes  a  short-lived,  but 
significant,  contribution  during  the  attack  period,  but  then 
is  negligible.  Personnel,  graphed  in  Figure  5.63,  is  a 
strr--  contributor  primarily  during  the  first  15  days.  Both 
are  important  if  the  war  will  last  a  short  time  and/or  if  the 
value  of  early  sorties  outweighs  that  of  later  sorties.  AIS 
makes  a  small  but  consistent  daily  contribution  after  Day  10 
(see  Figure  5.62).  Similarly,  Missiles  contribute  after  Day 
15.  These  observations  lead  to  a  rank-ordering  of  the 
factors  based  on  importance  of  contributions  to  sortie 
generation . 

Rank  order  based  on  Table  5.1  and  Figures  5.62  and  5.63 
is  the  followina: 

1.  Filler  aircraft 

2 .  Support  equipment 

3.  Spare  parts 

4.  Recovery  resources 

5.  Maintenance  personnel 

6.  AIS  test  sets 

7 .  Missiles 

8.  A3 DR  capability 
Fuel 

l 

i 
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This  assessment  is  subjective  and  takes  no  account  of  the 
cost  of  each  resource  area  in  terms  of  procurement,  training, 
and  maintenance. 

No-Attack  Case 

As  we  did  for  the  attack  case,  the  factors  are  ranked  in 
Table  5.1  according  to  the  expected  net  additional  sorties 
over  thirty  days  for  the  no-attack  case.  For  consistency, 
the  same  groupings  of  contributors  as  in  the  attack  case  are 
used  to  construct  Figures  5.64  and  5.65  to  show  comparatively 
which  factors  contribute  the  most  per  day  over  time.  From 
Figure  5.64,  it  is  obvious  that  Filler  Aircraft,  as  in  the 
attack  case,  is  the  dominant  factor  across  the  period. 
Although  the  contribution  of  Missiles  is  limited  to  short 
periods  of  time,  the  magnitude  of  the  expected  contribution 
is  very  significant.  As  a  result,  this  factor  is  considered 
the  second  most  important  for  the  no-attack  case.  (NOTE: 
Since  the  magnitude  of  the  missile  contribution  compresses 
the  y-scale  in  Figure  5.65,  this  figure  is  redrawn  as  Figure 
5.66  without  Missiles  so  the  remaining  factors  can  be 
compared.)  Similarly,  Fuel  is  ranked  as  the  third  most 
important  factor  due  to  its  large  potential  impact  on  sorties 
flown . 

A.s  in  the  attack  case,  Personnel  contributes  positively 
and  significantly  for  the  first  15  days.  3ecause  of  the 
potential  value  of  those  early  sorties.  Personnel  is  ranked 
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No-Attack  Case  —  Overall  Contribution  of  Five  Factors  (2) 
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Contribution  of  Five  Factors 
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Figure  5.56 

No-Attack  Case  —  Overall  Contribution  of  Five  Factors  (2) 


^•1 


216 


third  for  the  no-attack  case.  Spares  are  ranked  next  with 
generally  positive  contributions  throughout  the  period. 
Support  Equipment  contributions  are  rather  small  throughout 
the  period,  but  are  consistently  positive  except  during  the 
first  two  and  last  five  days.  Differences  between  ABDR  and 
AIS  are  difficult  to  distinguish,  while  Recovery  is  rated 
last  since  it  is  not  needed  in  the  absence  of  air  base 
attacks . 

Rank  order  for  the  no-attack  case,  based  on  Table  5.1 
and  Figures  5.64  -  5.66,  is  the  following: 

1 .  Filler  aircraft 

2.  Missiles 

3.  Fuel 

4 .  Maintenance  personnel 

5.  Spare  parts 

6.  Support  equipment 

7.  ABDR  capability 

8.  AIS  test  sets 

9.  Recovery  resources 

As  in  the  attack  case,  this  assessment  is  subjective  and 
takes  no  account  of  the  cost  of  each  resource  area  in  terms 
of  procurement,  training,  and  maintenance. 


217 


CHAPTER  VI  -  CONCLUSIONS  AND  FUTURE  RESEARCH 

From  the  many  results  presented  in  the  two  previous 
chapters,  we  now  draw  some  overall  conclusions  which  may  be 
useful  to  logistics , managers  and  researchers.  Before 
discussing  specific  conclusions,  we  first  define  the  scope  of 
this  research.  Next,  conclusions  for  each  research  objective 
are  presented  based  on  the  results  for  that  objective.  Then 
we  address  some  limitations  of  the  study  and  finally 
recommend  some  areas  for  future  research. 

Scooe 

Conclusions  must  be  within  the  scope  of  this  research. 
The  design  of  the  experiment  limits  the  number  and  type  of 
possible  valid  conclusions  we  can  draw  from  the  results.  Our 
simulation  experiment  models  a  single,  specific  air  base  with 
one  type  of  aircraft,  the  F-15.  As  such,  conclusions  may  not 
be  extendable  to  other  air  bases  or  types  of  aircraft. 
Further,  although  we  use  eight  random  versions  of  a  set  of 
air  base  attacks,  the  attacks  are  constant  with  respect  to 
the  number  of  attackers,  their  desired  aimpoints  and 
approaches  to  the  air  base,  timing  of  the  attacks,  types  of 
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munitions  used,  etc.  A  change  in  any  of  these  factors  could 
influence  the  results  and  conclusions. 

A  very  important  assumption  imbedded  in  the  entire  work 
concerns  the  TSAR  and  TSARINA  simulation  models.  We  have 
assumed  that  the  processes  modeled  by  these  simulations  are 
accurate  and  realistic  representations  of  the  real  world. 
Actually  these  models  are  also  metamodels  themselves  of  an 
air  base's  logistics  infrastructure  placed  in  a  hostile 
environment.  Thus  conclusions  are  contingent  upon  the 
processes  as  modeled  by  TSAR  and  TSARINA.  Given  this  scope, 
we  now  present  some  conclusions  drawn  from  the  results  of 
this  research. 


Part  of  this  objective  concerns  variance  reduction 
through  the  use  of  a  statistically  controlled  experiment. 

The  idea  is  to  use  techniques  which  reduce  the  variance  of 
the  output  random  variable  from  the  simulation.  If  this  can 
be  dcr.e  without  disturbing  its  expected  value,  we  can  obtain 
either  greater  precision  from  a  fixed  number  of  runs,  or  the 
same  level  of  precision  from  fewer  runs  (Law  and  Kelton, 
1932) .  This  can  be  significant  especially  when  using  large, 
exper.sive-to-run  simulation  models.  A  second  part  of  this 
objective  concerns  designing  the  experiment  in  such  a  way 
that  we  can  make  statistically  valid  hypothesis  tests  and 
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comparisons  of  interest.  The  information  we  want  to  glean 
from  the  simulation  data  must  be  considered  in  the  design  of 
the  experiment . 

Conclusion  1:  Good  variance  reduction  results  are 
obtained  through  the  use  of  common  random  numbers  applied  to 
a  classical  two-level  fractional  factorial  design  based  on 
blocking  within  the  fractional  replication.  The  very  large 
size  of  the  TSAR  and  TSARINA  models  and  the  scope  of  the 
problem  make  the  variance  reduction  significant.  Additional 
replications  of  a  completely  randomized  design  which  would 
give  the  same  precision  as  our  blocking  design  would  cost 
about  480  additional  minutes  (almost  8  hours)  of  computer 
time  on  the  NP-1  Gould  supercomputer.  This  breaks  down  to 
402  minutes  for  additional  replications  of  the  attack  case 
and  78  minutes  for  the  no-attack  case.  Further,  no 
extraordinary  time  or  effort  is  necessary  to  apply  the 
variance  reduction  technique  of  common  random  numbers  to  the 
TSAR/TSARINA  models.  Rather  the  blocking  scheme  is  easy  to 
use  and  fits  naturally  into  the  design  of  the  experiment.  In 
fact,  the  design  is  a  standard  fractional  factorial  available 
in  factorial  design  bocks.  Thus  this  "real-life"  application 
is  a  good  example  of  where  variance  reduction  techniques 
yield  good  results  and  reveal  that  such  techniques  may  be 
very  useful  even  in  large-scale,  very  complex  simulation 
experiments  . 
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Conclusion  2:  Our  results  appear  to  confirm  that  the 
variance  of  daily  sorties  flown  is  not  homogeneous  between 
the  attack  and  no-attack  cases,  thus  justifying  the 
innovation  of  running  the  experiment  as  two  sub-experiments. 
This  structure  allows  comparison  of  like  design  points  for 
the  two  cases  where  we  test  the  hypothesis  of  no  difference 
in  the  daily  mean  response  and  estimate  confidence  intervals 
for  the  difference..  The  use  of  common  random  numbers  induces 
correlation  between  the  two  cases  and  results  in  a  smaller 
variance  for  the  difference  in  mean  responses  for  a  given 
design  point, 

Ressarsh  Objective  2; _ Estimating  Metamodels 

This  objective  involves  the  estimation  of  simpler 
metamodels  from  the  simulation  results  so  that  sorties  can  be 
predicted  based  on  resource  inputs.  A  metamodel  is  estimated 
for  each  day  and  includes  significant  main  effects  and  two- 
way  interactions . 

Conclusion  1:  Two-way  interactions  between  the  main 
effects  are  extremely  important.  While  other  research  has 
shown  the  importance  of  the  main  effects,  this  research  shows 
that  two-way  interactions  are  just  as  important  in  explaining 
sortie  performance  and  should  not  be  excluded  in  analyses. 

Conclusion  2:  As  evident  from  the  results,  the 
logistics  infrastructure  and  the  generation  of  sorties 
involves  many  complex  and  interdependent  relationships.  One 
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particular  observation  is  the  apparent  cost  of  flying  early 

in  the  30-day  time  period  at  the  expense  of  later  sorties. 

Many  incidents  occur  where  the  high  level  of  a  resource  or 

interaction  term  appears  to  allow  the  flying  of  additional 

sorties  in  the  early  period;  however  that  same  high  level 

seems  to  result  in  fewer  sorties  flown  (i.e.,  negative  beta 

coefficients  in  the  predictor  equation)  in .later  daily 

models.  Thus  we  conclude  that  one  resource  level  may  be 

sufficient  for  immediate  capability,  but  a  different  level 

may  be  necessary  and  more  beneficial  for  sustained 

capability.  Both  the  readiness  (i.e.,  immediate  capability) 

and  the  sustainability  of  the  logistics  infrastructure  are 

important.  However,  readiness  does  not  appear  to  necessarily 

lead  to  sustainability  and  vice  versa.  This  could  be  very 

important  with  a  fixed  budget  and/or  planning  horizon. 

Conclusion  3:  The  air  base  logistics  infrastructure  as 

modeled  here  still  contains  much  unexplained  variance  as 

2  2 

evidenced  by  the  R  results.  The  average  R  for  the  attack 
case  is  0.6365  while  the  no-attack  case  averages  0.6789. 
Either  the  problem  has  a  high  degree  of  inherent  variability, 
seme  other  important  factors  are  omitted,  or  some  higher- 
order  interactions  are  not  negligible. 

Research  Objective  3: _ Impact  of  Attacks 

The  experimental  design  is  structured  so  that  the  impact 
of  attacks  on  the  sortie  generation  effort  can  be  estimated. 


Here  we  want  to  estimate  the  difference  that  attacks  make  on 
the  level  of  flying  attained  and  determine  whether  different 
resources  are  more  important  than  others  in  the  presence  of 
air  base  attacks. 

Conclusion  1:  When  attacks  on  the  air  base  are 
included,  different  daily  metamodels  result  than  for  the  no¬ 
attack  case.  The  attacks  cause  apparent  losses  of  resources 
which,  in  turn,  degrade  sortie  generation.  This  results  in 
those  resources  becoming  significant  factors  in  the  attack 
case  whereas  they  may  not  be  significant  in  the  no-attack 
case.  Conversely,  since  the  attacks  slow  flying  and  hence 
the  use  of  consumable  resources  (e.g.,  missiles  and  fuel), 
these  resources  become  less  important  than  in  the  no-attack 
case  where  higher  sortie  rates  are  possible  and  lead  to 
higher  resource  consumption. 

Conclusion  2:  The  attacks  appear  to  significantly 
degrade  sortie  results  during  the  attack  period  regardless  of 
the  resource  posture  (except  when  Recovery  resources  are 
high) .  After  the  attacks  on  Days  1-5,  the  air  base  recovers 
to  some  maximum  level  of  sortie  production  per  day  which  then 
decreases  almost  linearly  over  time.  For  the  situation  where 
ail  resource  factors  are  high,  this  recovery  capability  is 
about  ISO  sorties  per  day  compared  to  about  150  sorties  per 
day  when  all  factors  are  low  (see  Figure  6.1.  Note:  Figure 
5.9  is  repeated  here  as  Figure  6.1) .  The  resource  posture 
seems  to  have  more  bearing  on  that  maximum  recovery  level  cf 
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sorties  per  day  than  the  attacks  themselves.  As  seen  in 
Figure  6.1,  sortie  generation  appears  to  degrade  by  5-7 
sorties  per  day  once  the  air  base  has  recovered  from  the 
attacks.  If  the  resources  in  place  on  Day  1  can  be 
adequately  protected,  the  long-term  effect  of  the  attacks  may 
be  minimized  although  we  lose  a  significant  number  of  sorties 
during  the  attack  period  due  to  runway  closures. 

Research  -Objective  4: _ Key.-Rssou.r.C..ea 

The  rank-order  results  for  the  separate  cases,  attack 
and  no-attack,  are  next  synthesized  to  subjectively  rank  the 
factors  with  regard  to  potential  sortie  contributions  over 
the  thirty  day  period  when  the  likelihood  of  attack  is 
unknown.  If  a  factor  is  found  to  be  relatively  unimportant, 
this  is  based  on  a  comparison  of  the  high  and  low  levels,  not 
on  the  absence  of  the  factor  altogether.  Thus  some  basic 
capability  is  assumed  to  be  important  for  all  th '  factors. 

The  factors  are  discussed  below  in  rank  order,  most  important 
to  least  important. 

Filler  aircraft  are  the  surest  way  to  increase  sortie 
production  with  or  without  air  base  attacks.  This  is  logical 
since  extra  aircraft  keep  the  pool  of  aircraft  potentially 
available  to  fly  at  a  high  level.  Whether  or  not  aircraft 
will  be  available  as  replacements  during  a  war  is  another 
question  due  to  the  expense  of  aircraft  (and  limited  budgets) 
plus  the  lengthy  procurement  pipeline.  Potential  sources  of 
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filler  aircraft  include  US  Air  Force  Reserve  and  Air  National 
Guard  assets  which  usually  mobilize  after  active-duty  units. 

Spare  parts  are  ranked  second  overall  due  to  the 
consistent  contributions  to  sortie  production  in  both  cases. 
Spares,  like  aircraft,  are  expensive  and  the  pipelines  for 
procurement  and  repair  are  long.  A  possible  action  to 
consider  is  the  adequate  protection  of  existing  resources 
from  attacks  since  spares  are  very  vulnerable  and  sensitive 
to  attacks  on  the  air  base. 

Missiles  and  fuel  are  jointly  ranked  third  due  to  their 
importance  and  similarity  in  meeting  high  sustained  sortie 
rates.  Both  are  consumables  which  must  be  available  to  meet 
the  high  demands  for  sorties.  While  both  did  not  appear  to 
be  susceptible  to  air  base  attacks  in  the  modeling  done  in 
this  research,  they  are  resources  which  must  be  resupplied. 
Deliveries  are  potentially  very  vulnerable  to  interception, 
sabotage,  and  destruction.  Sufficient  protection  and  "safe" 
modes  of  transportation  ought  to  be  considered  for  these 
resources . 

Support  ecru  foment  proved  to  be  a  very  large  contributor 
in  the  presence  of  attacks,  thus  suggesting  that  important 
assets  are  lost  in  the  attacks.  Since  the  low  resource  level 
seems  sufficient  to  support  flying  in  the  no-attack  case, 
better  protection  in  case  of  attack  might  be  adequate  rather 

The  potential  gains  are  high  if  these 


than  more  equipment . 


226 


S 


AIS  test  set  (i.e.,  having  an  additional  set)  makes 
little  contribution  in  the  no-attack  case,  but  consistently 
provides  added  sorties  in  the  attack  case  after  Day  10.  Thus 
it  seems  that  the  impact  of  the  attack  on  either  spare  parts 
or  the  AIS  itself  increases  the  value  of  having  the  second 
set  of  AIS.  Due  to  the  expense  and  difficulties  in 
maintaining  the  AIS,  additional  sets  may  not  be  realistic. 
Alternatively,  we  ought  to  ensure  adequate  protection  of 
available  resources. 

Bec.P-vary_  resources  .and  maintenance  Personnel  reflect 
resources  that  provide  significant  additional  sorties  in  the 
early  days  of  the  war.  Since  the  war  may  be  won  or  lost  in 
the  first  days  of  hostilities,  these  sorties  may  be  extremely 
important  and  outweigh  the  value  of  later  sorties.  As  with 
most  of  the  resources,  procurement  can  be  expensive  and 
training  can  be  long. 

A3  DP.  capability  contributes  the  least  of  all  the  factors 
when  both  cases  are  considered.  This  may  simply  mean  that 
the  low  level  is  sufficient  and  the  high  level  of  capability 
is  not  necessary. 


Several  limitations  ~re  evident  in  this  research. 
First,  the  value  of  sorties  is  assumed  to  be  erual  and  tim; 
independent .  However,  early  sorties  may  actually  be  more 
and  influential  in  determining  the  outcome  of  the 
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war.  Also  different  types  of  flying  missions  may  have 
different  values.  We  do  not  address  any  difference  in  sortie 
values . 

A  second  limitation  is  that  the  cost  of  resources  is  not 
considered.  Increasing  the  level  of  one  resource  may  lead  to 
more  sorties,  but  may  be  prohibitively  expensive  or  less 
cost-effective  than  increasing  another  resource. 

Although  this  research  broadens  the  inference  space  of 
previous  research  approaches,  there  are  still  limitations  to 
the  conclusions  we  can  draw.  Since  the  independent  variables 
are  qualitative,  we  can  predict  only  at  the  low  and  high 
levels,  thus  missing  any  resource  or  capability  positions  in 
between.  Also,  although  we  have  included  random  versions  of 
an  optimized  attack,  there  are  still  countless  other  attacks 

t 

which  are  not  considered.  This  further  limits  the  inference 
space  when  the  metamodels  are  used  to  predict  performance. 

A  final  limitation  is  the  modeling  of  thirty  separate 
metamodels,  one  for  each  day.  A  measure  which  spans  the  time 
period  of  interest  might  prove  beneficial  in  comparing 
various  resource  postures.  These  limitations  provide  several 
areas  for  future  research. 

r*  II  U  !T  £  P.0S0S  —  c.'n 

Several  areas  for  continued  work  beyond  the  limitations 
mentioned  above  are  possible.  Topical  areas  are  variance 
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reduction,  widening  of  the  inference  space,  and  performance 
measures . 

Variance  Reduction 

Only  one  variance  reduction  technique,  i.e.,  common 
random  numbers,  is  used  in  this  research.  Although  we 
obtained  good  results,  there  is  ample  room  for  further 
improvements.  Another  technique,  the  use  of  control 
variables',  may  improve  the  results  found  here.  A  control 
variable  is  a  random  variable  that  is  correlated  with  the 
response  variable  Y  and  we  know  its  expected  value.  Thus  a 
random  variable  C  is  a  control  variable  of  Y  if  it  is 
correlated  with  Y  and  we  know  jic .  Lavenberg  and  Welch  (1981) 
provide  the  following  development  of  how  the  control  variable 
C  is  used  to  construct  an  unbiased  estimator  for  |l  which  has 
a  smaller  variance  than  the  unbiased  estimator  Y  where  E(Y)  = 
[i.  For  any  constant  b, 

Y  (b)  =  Y  -  b  (C  -  ^c) 
is  also  an  unbiased  estimator  of  p..  Now 

Var [ Y (b) ]  =  Var(Y)  -  2b  Cov(Y,C)  +  b2Var(C). 

2b  Co v ( Y , C )  >  b2Var (C) 

then  Y (b)  has  a  smaller  variance  than  Y.  The  value  cf  b 
which  minimizes  Var[Y(b)]  is 
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B  =  Cov(Y,C)  /  Var(C) 
and  the  resulting  minimum  variance  is 

Var [Y  (B) ]  =  (1  -  pyC)  Var(Y) 

2 

where  pYC  is  the  correlation  coefficient  between  Y  and  C. 

Thus  the  greater  the  correlation  between  C  and  Y,  the  greater 
the  variance  reduction. 

Lavenberg  and  Welch  (1981)  also  identify  the  two  key 
problems  in  applying  control  variables:  a)  finding  control 
variables  which  are  highly  correlated  with  the  estimators  of 
interest,  and  b)  estimating  the  optimum  coefficient  vector  3 
which  is  unknown.  Effective  control  variates  may  possibly  be 
isolated  in  TSAR  and/or  TSARINA;  additional  experimentation 
with  the  models  is  required  to  find  candidates.  Some 
possibilities  include  aircraft  break  rates  and  attack 
attributes  such  as  number  of  attackers,  number  of  bombs 
dropped,  etc.  Break  rates  reflect  how  well  the  aircraft  are 
operating  and  seem  logically  related  to  the  number  of  sorties 
flown,  i . e . ,  as  the  break  rate  decreases,  we  expect  to  fly 
more  sorties.  We  also  ought  to  be  able  to  calculate  the 
expected  break  rate  from  the  probabilities  of  failure  for  the 
81  different  systems/subsystems  modeled  by  TSAR.  Concerning 
the  attacks  on  the  air  base,  we  would  expect  mere  damage  and 


hence  less  flying  if  the  number  of  attackers  or 
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bombs  dropped  increases.  Both  of  these  factors  are  inputs  to 


the  experimenter. 


SARINA  and  under  the  direct  control  of 
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Also  the  structure  of  the  model  allows  easy  computation  of 
the  expected  value  of  both.  Future  experimentation  is 
necessary  to  understand  how  these  control  variable  candidates 
are  treated  within  the  simulation  and  to  discover  additional 
ones . 

Further  improvement  in  the  variance  reduction  results 

might  be  realized  with  a  modification  to  TSAR  that  isolates 

the  random  number  streams  used  in  the  model.  Streams  could 

possibly  be  isolated  by  type  of  activity  and  this  ought  to 

allow  more  commonality  in  the  use  of  random  numbers  between 
design  points.  For  example,  random  number  stream  one, 

generated  by  random  seed  1^,  could  be  used  to  check,  whether 

or  not  an  aircraft  is  attrited,  R2  (generated  by  random  seed 

I2)  could  be  used  to  check  whether  or  not  there  is  battle 

damage,  R^  (generated  by  random  seed  I^)  could  be  used  to 

check  whether  or  nor  system  x  failed,  etc.  Each  R^  could  be 

initialized  with  a  random  seed  I ^  and  thus  we  would  have  a 

seed  vector 

~0  *  *0  ^  '  '**' 

which  represents  a  common  random  number  secuence  to  be  used 
by  a  set  or  block  of  design  points  in  the  experimental 
design.  Due  to  the  complexity  of  the  TSAR  simulation, 
complete  congruency  of  random  number  streams  is  difficult  at 
best;  nowever  this  suggested  modification  should  allow  a  much 
"cleaner"  application  of  the  correlation  induction  technique 
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of  common  random  numbers  which  ought  to  lead  to  better 
variance  reduction  results. 


Widening  the  Inference  Space 

The  inference  space  can  be  expanded  by  allowing  more 
levels  within  the  resource  structure  of  the  experimental 
design  and  by  diversifying  the  attack  scenario.  Some 
suggested  approaches  are  discussed  below. 

The  'experimental  design  in  this  research  is  a  two-level 
fractional  factorial  which  attempts  to  bound  the  problem 
between  the  most  likely  high  and  low  resource  positions  for 


each  resource  variable.  While  this  may  bound  the  inference 
space,  it  doesn't  help  the  decision-maker  using  the  estimated 
metamodels  when  the  actual  resource  levels  are  somewhere  in 
the  middle;  thus  he/she  must  select  either  the  high  or  the 
low  as  the  closer  or  more  representative.  As  a  result,  we 
potentially  lose  seme  precision  in  the  true  estimated 
capability  of  the  logistics  infrastructure.  Further, 
previous  experience  with  the  model  and  the  logistics 
environment  indicate  that  there  may  be  a  threshold  or  "knee" 
in  the  curve  as  one  moves  from  the  low  level  to  the  high  for 
many  of  the  resources .  With  this  in  mind,  further  research 
with  a  three  level  design  would  be  valuable. 

In  a  three  level  experiment,  we  would  add  a  middle  or 
medium  level  to  the  high  and  low  levels  used  in  the  two  level 
design.  Determining  that  middle  level  requires 
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experimentation  with  TSAR  to  determine  the  appropriate  value 
for  each  factor  or  resource.  Connor  and  Zelen  (1959)  provide 
examples  of  fractional  factorial  designs  for  three  level 
experiments.  Their  Plan  243.10.27  on  page  37  is  a  1/243 
replication  of  10  factors  in  9  blocks  of  27  observations  each 
for  a  total  of  243  observations  per  case  (attack  and  no¬ 
attack)  or  a  total  of  586  simulation  runs  of  30  days  each. 

All  two-factor  interactions  except  two  are  measurable  with 
this  design.  Although  this  three  level  experimental  design 
more  than  doubles  the  amount  of  simulation  required  by  our 
two  level  design,  the  increased  precision  in  predictions  of 
the  metamodels  and  the  insights  gained  by  decision  makers  may 
be  well  worth  the  expense. 

Another  research  area  concerning  the  expansion  of  the 
inference  space  deals  with  the  hostile  environment, 
specifically  the  attacks  on  the  air  base  as  modeled  by 
TSARiNA.  Several  issues  require  more  research.  One  area 
concerns  the  variability  of  the  attack  results  generated  by 
TSARINA.  Folkeson  (1988)  indicates  that  great  variability  in 


results  occurs  when  the  TSARINA  random  number  seeds  change, 
r.e  claims  that  variance  does  not  stabilize  until  about  1000 
triais  have  been  run.  For  research  such  as  this  thesis,  1000 
^rla-s  -s  prohibitive;  therefore  more  research  is  necessary 
uO  understand  the  at— ack  results  of  TSARINA  as  influenced  bv 
tr.e  choice  of  random  seed.  Then,  given  this  understanding, 
we  need  to  know  how  to  better  ir.corocrate  these  attack 
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results  in  research  concerning  the  sensitivity  of  sortie 
performance  to  the  randomness  of  the  attacks.  For  example, 
in  this  thesis,  we  used  eight  blocks  where  each  block  (in  the 
attack  case)  represented  a  random  version  of  the  same  attack. 
Would  16  blocks  have  been  more  appropriate  given  the 
randomness  due  to  TSARINA,  or  would  less  have  been 
sufficient?  To  answer  these  questions  more  research  and 
experimentation  with  TSARINA  is  needed. 

Another  issue  needing  research  is  the  effect  of 
different  types  of  attack  on  the  air  base.  It  would  be 
useful  to  a  decision  maker  at  an  air  base  to  have  some 
insights  on  how  different  types  of  attacks  affect  flying 
performance.  Here  the  difference  needs  to  be  in  terms  of 
numbers  of  attackers,  aim  points,  munitions  used,  etc.  rather 
than  just  randomness  of  an  attack  where  these  factors  are 
fixed.  A  possible  approach  might  be  where  the  blocks  are 
different  types  of  attacks  rather  than  random  versions  of  the 
same  attack.  This  would  broaden  the  inference  space  and 
provide  a  more  general  picture  of  the  capability  of  the 
logistics  infrastructure  within  an  uncertain  hostile 
environment . 

Rerfc_r~ar.ee  Measures 

Research  potential  is  great  in  the  area  of  measures  of 
performance  over  time.  Possible  approaches  are  discussed 
below  for  mulcivariate  responses  and  time  series  analysis. 
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One  of  the  above  conclusions  (Research  Objective  2, 
Conclusion  2)  is  that  resource  postures  that  provide  a  high 
level  of  sorties  in  the  early  days  do  not  necessarily  lead  to 
a  high  level  of  sorties  later  in  the  time  period.  Similarly, 
a  posture  that  leads  to  a  high  level  of  flying  in  the  later 
days  may  not  provide  or  support  very  many  sorties  in  the 
early  days.  Thus  we  have  a  potential  conflict  between 
immediate  capability  or  readiness  and  sustained  capability. 
Each  of  these  two  capability  objectives  may  require  a 
different  set  of  resources  and  thus  tradeoffs  are  necessary. 
This  leads  to  a  reconsideration  of  a  measure  of  performance 
that  is  time  dependent. 

One  approach  might  be  to  use  a  utility  function  which 

assigns  weights  to  the  daily  sorties  flown  and  then  sum  the 

weighted  values  over  time  to  derive  a  single  response.  Thus 

for  our  thirty  daily  responses,  we  can  develop  a  single 

objective  function 

Y  =  wlYl  +  w2y2  +  . . .  +  w30v30 

where  v,  is  the  number  of  sorties  flown  on  Dav  i  and  w,  is 

the  appropriately  chosen  weight  for  the  sorties  flown  on  Day 
i.  The  problem  with  this  approach  is  determining  the  weights 
which  is  highly  subjective  and  highly  dependent  on  the 
environmental  scenario. 

A  second  approach  is  to  use  multiple  responses  where  the 
(k) 

response  Y  for  a  design  point  k  is  the  vector  of  daily 
sorties  flown 
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Y(k)  =  [Y;l  Y2 


L30 


] 


and  we  use  multivariate  multiple  linear  regression  methods  to 
estimate  a  metamodel.  An  alternative  approach  is  to  divide 
the  responses  into  time  periods  of  interest  such  as  the  first 
five  days  where  immediate  capability  is  important  and  then 
the  days  after  that  where  sustained  capability  is  more  of  a 
concern.  For  example,  given  that  y^  is  the  number  of  sorties 
flown  on  Day  i,  we  might  let 


(k) 


-  2  Yi 

i=l 


(k) 


30 

=  S  Yi 
i=6 


and  the  response  of  interest  is 


Y<k)  =  [v(k)  yik)  ] 


and  we  perform  our  multivariate  estimation  of  the  metamodel. 

Useful  approaches  to  this  research  problem  may  also  be 
found  in  the  field  of  time  series  analysis.  Jenkins  (1979) 
describes  rive  classes  of  time  series  models  which,  with 
ruture  research,  could  be  applied  to  the  sortie  generation 
formulation  described  here.  Based  on  our  conclusions,  two  of 
tne  classes  of  models  are  particularlv  relevant.  First  is 
tne  multivariate  stochastic  model  which  assumes  feedback 
between  output  and  inputs.  As  our  results  indicate,  resource 
revels  help  determine  the  number  of  sorties  flown,  but 
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sorties  flown  also  affect  resource  levels.  Modeling  the 
feedback  between  sorties  and  resources  may  help  explain  why 
high  resource  levels  can  lead  to  high  sorties  flown  early  in 
the  time  period  of  interest,  but  also  result  in  fewer  sorties 
in  later  days.  Figure  6.2  is  an  adaptation  of  a  figure  in 
Jenkins  and  Alavi  (1981)  showing  the  feedback  relationship 
between  two  variables .  The  idea  is  that  the  variables  are 
treated  on  an  equal  or  reciprocal  basis  to  be  able  to 
describe  the  mutual  dependence  between  them. 

A  second  class  of  time  series  models  which  future 
research  should  explore  is  intervention  models. 

Interventions  are  abnormal  events  or  effects  which  are  not 
easy  to  quantify;  dummy  variables  are  often  used  to  represent 
such  behavior  either  as  pulse  or  step  variables  (Jenkins 
1979) .  In  cur  problem,  attacks  on  the  air  base  could  be 
modeled  as  interventions  or  interruptions  in  the  time  series 
representing  sorties  flown.  Box  and  Tiao  (1975)  describe 
responses  to  step  and  pulse  inputs.  Figure  6.3  presents  a 
possible  response  based  on  the  results  of  this  research  where 
attacks  seem  to  cause  an  initial  degradation  in  sorties  flown 
from  which  the  air  base  recovers  to  a  level  dictated  by  the 
resource  posture  of  the  logistics  infrastructure.  Lewis-Beck 
(1986)  used  a  similar  idea  with  interrupted  time  series 
analysis.  He  presents  a  least  squares  procedure  which  looks 
i-induced  changes  in  the  mean  a 


for  interventicr 


nd/or  slope  of 


Initial  Response  =  co1  +  co2 


Final  Effect  = 

2 


Figure  6.3 

Response  to  an  Attack  as  an  Intervention 
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a  time  series.  These  approaches  have  much  potential  for  the 
modeling  of  air  base  attacks. 

Lastly,  response  surface  methodology  or  some  type  of 
opt imizat ion-through-simulation  could  be  applied  to  find  the 
optimal  combination  of  resources  to  maximize  sortie 
generation.  Here,  the  problem  would  be  approached  as  a 
planning  or  resourcing  problem  to  outfit  an  air  base  .to 
maximize  performance  with  cost  constraints.  Research  in  this 
area  is  an  almost  entirely  new  approach  to  the  air  base 
problem  described  here  rather  than  a  straight-forward 
extension  of  our  research.  However,  many  of  the  same  issues 
will  have  to  be  addressed,  such  as  experimental  design  and 
variance  reduction,  for  a  successful  application  of 
optimization  theory. 
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Appendix  A: _ Equivalency  of  Model  Forms 

Let  us  consider  a  simple  model  with  two  factors  at  two 
levels  each  and  an  interaction  term.  Two  coding  schemes  for 
the  factor  levels  can  be  used.  In  Model  1  we  use  (0,1)  to 
respectively  denote  the  low  and  high  level  of  each  factor. 
The  form  of  the  model  is: 

Y  =  (3o  +  Mi  +  p2X2  +  P12X1X2 
and  the  various  combinations  of  factor  levels  gives  the 
following  equations  for  the  dependent  variable: 


X 


1 


=  0 


Xi  = 


1 


X 


2 


X 


2  " 


Po 

po+  ^1 

3+3 
p0  H2 

3q+  3x+  P2+  312 

A  second  coding  scheme  uses  (-1,1)  to  respectively 
denote  the  low  and  high  level  of  each  factor.  The  form  of 
Model  2  is: 

Y  =  p.  +  a  +  j3  +  y 

and  the  various  combinations  of  factor  levels  gives  the 
following  equations  for  the  dependent  variable: 
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$2  Pl2 
(7)  p  =  f-  +  *r 


Now  we  can  solve  (1)  for  y  by  substituting  (5),  (6),  and  (7) 

for  values  in  (1) : 

(1)  p0  =  M--cx-P  +  Y 

Bi  ?  Bi  Bi  ?  B?  3i2 

Y 


r  v  r  *  • 

Po  -  <Po  ^  ^  4  +  ¥>  + 


Pl2 

(8)  Y  =  — 


From  this  simple  example  we  can  see  that  we  can  express 
the  equivalency  between  the  two  models  and  coding  schemes. 

It  is  also  apparent  that  the  computations  can  be  very 
extensive  to  express  this  equivalency  for  ten  factors 
including  all  two-way  interactions. 
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Appendix  B:  Regression  Results  for  No-Attack  Case 
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Appendix  C;  Residual  Results  for  No-Attack  Case. 
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The  son  of  a  career  Air  Force  officer, 
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received  a  Ph.D.  in  Management  from  Purdue  University  in 
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